Binance Data Scraping Script 2019. All pairs, data since launch of Binance, automatically builds MACD, EMA, Bolls, RSI and MAs [Stadivm]

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
71
Reaction score
8
Location
Stadivm.com
Hello & Welcome to anyone new to Stadivm.


So what is this all about? Well what I've uploaded here is a group of scripts I was using for gathering historical data from Binance's API, and then also building a load of technical indicators I was using to do backtesting of algorithmic trading strategies.

Actually these scripts make calls to the Binance API, get data from all pairs going back to the launch of the platform, join together the results chronologically, and then process the raw data to build indicators like MACD, RSI, Moving averages, EMAs and Bollinger Bands.


These are the default indicators which are created:
  • MACD (12, 26, 9)
  • EMA (12 26)
  • Bollinger Bands (20 and 36)
  • RSI (14)
  • Moving Averages (7, 25, 99)
  • And a custom indicator I created, "Volume Over Averages (VOA)" (9, 27, 81)
I'm by no means an expert python programmer or expert algo trader, but this was the 3rd version of these scripts and after ironing out some glitches in the first two versions this one did me proud over the time I was using it to gather data.


So as a heads up, this software is my own creation, owned completely by me and I give anyone that wants it full and unrestricted licensing to use this code in any way they would like including any form of commercial use including providing it for sale. I'm no longer using this code, and now it's yours to do with it as you please.

But hey, also please bear in mind that I also am providing this code openly on the proviso that it's being given to you as-is, and that by using this code you agree that it is under your own free will and that no liability for the way that you use it or the results of using it are to be placed upon myself personally or the Stadivm platform.



**

Don't forget to sign up to Stadivm to learn more about crypto algo trading, and to be a part of a growing community of tech enthusiasts who focus on different ways to create an income from future technologies like blockchain, AI and VR.

**

Steps to setting up the bot - what you need to do:


1.
Have your own Python environment setup and ready to use. I use Pycharm for all of my coding, but obviously any setup that works will work just fine. Here's a guide if you're new to Python programming and want to learn how to set up an environment: https://www.tutorialspoint.com/python/python_environment.htm


2. You'll need to install all of these Python modules:
  • requests
  • statistics
  • asyncio
  • pymongo
3. Setup up mongo db on your server/local system in order to store all of the data which is retrieved and processed from Binance. Here's some good guides on Mongo if you're new to it:
4. Download and install these scripts in a single directory (please see also that these scripts have been virus scanned using VirusTotal which you check out see below):

ZippyShare files:
https://www46.zippyshare.com/v/gJShNXjz/file.html

VirusTotal files:
https://www.virustotal.com/en/url/73bec26ee3f05209bb386dc51f14caf24e38adfae3cac82f59e9128f57b280d1/analysis/
https://www.virustotal.com/en/file/cbc4f7175a7b413c5d42d4147e36e85478f5a6094030e01efd432224ae0a80d2/analysis/1552308652/


5. Once those files are installed on your system, the last thing you need to do is to make some final adjustments to the code:

  • You can adjust the interval of the data from '1h' intervals to any of the other intervals that are on Binance (ie. 5m, 15m etc). This can be done at the bottom of the script "data_bot_v3.py" changing both of the instance of '1h' to whatever you would like instead.

  • You need to get a GUI for dealing with mongo db's called Robo 3T: https://robomongo.org/. Follow their setup guides and connect to their your remote or local system in order to be able to create new mongo databases for storing the Binance data.

  • You need to create a database called "binance_historic" and within it you need to create a collection called 'data1h' if you're gathering 1 hour data, or 'data5m' for example, if you're gathering 5 minute interval data.

  • In the mongo_bot.py file, you need to add your mongo db login details in the field at the top "mongo_uri". This authenticates your access to the mongo database from the python script automatically. As an example of the format:

    'mongodb://{mongo username}:{mongo password}@localhost:27017/?authSource=admin'

    Replace {mongo username} and {mongo password} with your details.
And that's it. Run time will vary depending on the exact parameters that you add and also the power of the system you're using, but typically it could take a few hours for me to build all of this data and have it stored.

Hit me up here or privately if you have any questions or are stuck on any part of getting this working, and I'll do what I can to work through the process with you.

If you're new to crypto algorithmic trading, feel free to post any questions you have in the 'Crypto Algo & Bot Trading' subforum (https://stadivm.com/forums/crypto-algo-bot-trading.101/), and myself or someone else with a bit of experience will do our best to help you out.
 
Last edited:
Reactions: IVOnoXx

IVOnoXx

Neophyte
Joined
Nov 18, 2018
Messages
7
Reaction score
4
This is great - would be good to see some more scripts like this for other exchanges as well and plotting data.
 
Reactions: Entelechy

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
71
Reaction score
8
Location
Stadivm.com
Sure we'll be releasing more scripts over the coming few days. They include:

- A script to gather a daily list of all new ICO's listed on the major platforms (goes through tens of thousands of listings to find the ones that have just been listed)

- A script to visualize the data gathered from the Binance API

- Considering also releasing a full script for a crypto backtesting engine as well.

Keep an eye on your alerts here so that you see when those scripts are released as well, or you can accept our push notifications also for updates. Thanks for your support!
 

bigboy87

Neophyte
Joined
Mar 15, 2019
Messages
2
Reaction score
1
Location
Vietnam
Reactions: Entelechy

deathslayer112

Neophyte
Joined
Mar 18, 2019
Messages
2
Reaction score
1
Location
Los Angeles
Dude, this is epic! Please let me know when you release this:

- A script to gather a daily list of all new ICO's listed on the major platforms (goes through tens of thousands of listings to find the ones that have just been listed)

I HAVE BEEN LOOKING FOR THAT EVERYWHERE!!!
 
Reactions: Entelechy