How to Get All Historic Data from Binance's API Using this Script, and Build MACD, RSI etc.

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
134


Hello & Welcome to anyone new to Stadivm!


So what is this all about? Well what I've uploaded here is a group of scripts I was using for gathering historical data from Binance's API, and then also building a load of technical indicators I was using to do backtesting of algorithmic trading strategies.

Actually these scripts make calls to the Binance API, get data from all pairs going back to the launch of the platform, join together the results chronologically, and then process the raw data to build indicators like MACD, RSI, Moving averages, EMAs and Bollinger Bands.


These are the default indicators which are created:
  • MACD (12, 26, 9)
  • EMA (12 26)
  • Bollinger Bands (20 and 36)
  • RSI (14)
  • Moving Averages (7, 25, 99)
  • And a custom indicator I created, "Volume Over Averages (VOA)" (9, 27, 81)
I'm by no means an expert python programmer or expert algo trader, but this was the 3rd version of these scripts and after ironing out some glitches in the first two versions this one did me proud over the time I was using it to gather data.


So as a heads up, this software is my own creation, owned completely by me and I give anyone that wants it full and unrestricted licensing to use this code in any way they would like including any form of commercial use including providing it for sale. I'm no longer using this code, and now it's yours to do with it as you please.

But hey, also please bear in mind that I also am providing this code openly on the proviso that it's being given to you as-is, and that by using this code you agree that it is under your own free will and that no liability for the way that you use it or the results of using it are to be placed upon myself personally or the Stadivm platform.



**

Don't forget to sign up to Stadivm to learn more about crypto algo trading, and to be a part of a growing community of tech enthusiasts who focus on different ways to create an income from future technologies like blockchain, AI and VR.

**

Steps to setting up the bot - what you need to do:


1.
Have your own Python environment setup and ready to use. I use Pycharm for all of my coding, but obviously any setup that works will work just fine. Here's a guide if you're new to Python programming and want to learn how to set up an environment: https://www.tutorialspoint.com/python/python_environment.htm


2. You'll need to install all of these Python modules:
  • requests
  • statistics
  • asyncio
  • pymongo
3. Setup up mongo db on your server/local system in order to store all of the data which is retrieved and processed from Binance. Here's some good guides on Mongo if you're new to it:
4. Download and install these scripts in a single directory (please see also that these scripts have been virus scanned using VirusTotal which you check out see below):

ZippyShare files:
https://www46.zippyshare.com/v/gJShNXjz/file.html

VirusTotal files:
https://www.virustotal.com/en/url/73bec26ee3f05209bb386dc51f14caf24e38adfae3cac82f59e9128f57b280d1/analysis/
https://www.virustotal.com/en/file/cbc4f7175a7b413c5d42d4147e36e85478f5a6094030e01efd432224ae0a80d2/analysis/1552308652/


5. Once those files are installed on your system, the last thing you need to do is to make some final adjustments to the code:

  • You can adjust the interval of the data from '1h' intervals to any of the other intervals that are on Binance (ie. 5m, 15m etc). This can be done at the bottom of the script "data_bot_v3.py" changing both of the instance of '1h' to whatever you would like instead.

  • You need to get a GUI for dealing with mongo db's called Robo 3T: https://robomongo.org/. Follow their setup guides and connect to their your remote or local system in order to be able to create new mongo databases for storing the Binance data.

  • You need to create a database called "binance_historic" and within it you need to create a collection called 'data1h' if you're gathering 1 hour data, or 'data5m' for example, if you're gathering 5 minute interval data.

  • In the mongo_bot.py file, you need to add your mongo db login details in the field at the top "mongo_uri". This authenticates your access to the mongo database from the python script automatically. As an example of the format:

    'mongodb://{mongo username}:{mongo password}@localhost:27017/?authSource=admin'

    Replace {mongo username} and {mongo password} with your details.
And that's it. Run time will vary depending on the exact parameters that you add and also the power of the system you're using, but typically it could take a few hours for me to build all of this data and have it stored.

Hit me up here or privately if you have any questions or are stuck on any part of getting this working, and I'll do what I can to work through the process with you.

If you're new to crypto algorithmic trading, feel free to post any questions you have in the 'Crypto Algo & Bot Trading' subforum (https://stadivm.com/forums/crypto-algo-bot-trading.101/), and myself or someone else with a bit of experience will do our best to help you out.
 
Last edited:

IVOnoXx

Neophyte
Joined
Nov 18, 2018
Messages
9
Reaction score
4
This is great - would be good to see some more scripts like this for other exchanges as well and plotting data.
 
Reactions: Entelechy

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
Sure we'll be releasing more scripts over the coming few days. They include:

- A script to gather a daily list of all new ICO's listed on the major platforms (goes through tens of thousands of listings to find the ones that have just been listed)

- A script to visualize the data gathered from the Binance API

- Considering also releasing a full script for a crypto backtesting engine as well.

Keep an eye on your alerts here so that you see when those scripts are released as well, or you can accept our push notifications also for updates. Thanks for your support!
 
Reactions: J.Babe.Abel

bigboy87

Neophyte
Joined
Mar 15, 2019
Messages
2
Reaction score
1
Location
Vietnam
Reactions: Entelechy

deathslayer112

Neophyte
Joined
Mar 18, 2019
Messages
2
Reaction score
2
Location
Los Angeles
Dude, this is epic! Please let me know when you release this:

- A script to gather a daily list of all new ICO's listed on the major platforms (goes through tens of thousands of listings to find the ones that have just been listed)

I HAVE BEEN LOOKING FOR THAT EVERYWHERE!!!
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
I am very new to all of this, but I forced myself to learn something new this month.

Followed all of your install and setup steps, the script does run, but hits a certain point and stops.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.binance.com', port=443'): Max retries exceeded with url: /api/v1/klines?symbol=ETHBTC&interval=5m&limit=1000&startTime.....

The error goes on.

Running Python37 within Windows 10.
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
I am very new to all of this, but I forced myself to learn something new this month.

Followed all of your install and setup steps, the script does run, but hits a certain point and stops.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.binance.com', port=443'): Max retries exceeded with url: /api/v1/klines?symbol=ETHBTC&interval=5m&limit=1000&startTime.....

The error goes on.

Running Python37 within Windows 10.
It could be a bunch of things, but as the error's saying it's getting a connection error and eventually failing because it's retrying unsuccessfully too many times.

Is it working fine before than and then throwing this?
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
It could be a bunch of things, but as the error's saying it's getting a connection error and eventually failing because it's retrying unsuccessfully too many times.

Is it working fine before than and then throwing this?
Yes, I think it starts(based on, it scrolls a lot of information for about 5-10min).

Here is a copy n paste of what I am observing:

Code:
{'open_time': 1500299400000, 'open': 0.083845, 'high': 0.08398, 'low': 0.083388, 'close': 0.08398, 'volume': 20.823, 'close_time': 1500299699999}, {'open_time': 1500299700000, 'open': 0.083767, 'high': 0.084195, 'low': 0.083699, 'close': 0.084086, 'volume': 13.58, 'close_time': 1500299999999}, {'open_time': 1500300000000, 'open': 0.084203, 'high': 0.08435, 'low': 0.083875, 'close': 0.08435, 'volume': 28.771, 'close_time': 1500300299999}, {'open_time': 1500300300000, 'open': 0.084298, 'high': 0.084428, 'low': 0.083678, 'close': 0.083678, 'volume': 29.747, 'close_time': 1500300599999}, {'open_time': 1500300600000, 'open': 0.08374, 'high': 0.083973, 'low': 0.083396, 'close': 0.083396, 'volume': 29.107, 'close_time': 1500300899999}, {'open_time': 1500300900000, 'open': 0.083568, 'high': 0.083896, 'low': 0.083239, 'close': 0.083686, 'volume': 29.174, 'close_time': 1500301199999}, {'open_time': 1500301200000, 'open': 0.083702, 'high': 0.084165, 'low': 0.083702, 'close': 0.083739, 'volume': 23.881, 'close_time': 1500301499999}, {'open_time': 1500301500000, 'open': 0.083821, 'high': 0.084561, 'low': 0.083688, 'close': 0.084462, 'volume': 29.684, 'close_time': 1500301799999}, {'open_time': 1500301800000, 'open': 0.084437, 'high': 0.085008, 'low': 0.084332, 'close': 0.084751, 'volume': 33.447, 'close_time': 1500302099999}, {'open_time': 1500302100000, 'open': 0.084922, 'high': 0.085843, 'low': 0.084922, 'close': 0.085665, 'volume': 19.015, 'close_time': 1500302399999}, {'open_time': 1500302400000, 'open': 0.08599, 'high': 0.086182, 'low': 0.085618, 'close': 0.085618, 'volume': 15.586, 'close_time': 1500302699999}, {'open_time': 1500302700000, 'open': 0.085499, 'high': 0.085505, 'low': 0.084791, 'close': 0.085076, 'volume': 20.076, 'close_time': 1500302999999}, {'open_time': 1500303000000, 'open': 0.085019, 'high': 0.08631, 'low': 0.085019, 'close': 0.08613, 'volume': 25.811, 'close_time': 1500303299999}, {'open_time': 1500303300000, 'open': 0.086106, 'high': 0.086106, 'low': 0.085627, 'close': 0.085798, 'volume': 18.701, 'close_time': 1500303599999}, {'open_time': 1500303600000, 'open': 0.086094, 'high': 0.08647, 'low': 0.085911, 'close': 0.086271, 'volume': 25.648, 'close_time': 1500303899999}, {'open_time': 1500303900000, 'open': 0.086707, 'high': 0.086707, 'low': 0.085472, 'close': 0.085584, 'volume': 23.209, 'close_time': 1500304199999}, {'open_time': 1500304200000, 'open': 0.085604, 'high': 0.086258, 'low': 0.085538, 'close': 0.086165, 'volume': 20.227, 'close_time': 1500304499999}, {'open_time': 1500304500000, 'open': 0.086366, 'high': 0.086366, 'low': 0.085372, 'close': 0.085372, 'volume': 18.517, 'close_time': 1500304799999}]
1499040000000
Traceback (most recent call last):
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\util\connection.py", line 80, in create_connection
    raise err
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\util\connection.py", line 70, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 301, in connect
    conn = self._new_conn()
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x000001DCEC652BE0>: Failed
 to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properl
y respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\util\retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url:
/api/v1/klines?symbol=ETHBTC&interval=5m&limit=1000&startTime=1499040000000 (Caused by NewConnectionError('<urllib3.conn
ection.VerifiedHTTPSConnection object at 0x000001DCEC652BE0>: Failed to establish a new connection: [WinError 10060] A c
onnection attempt failed because the connected party did not properly respond after a period of time, or established con
nection failed because connected host has failed to respond'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 473, in <module>
    data_bot = Data_Bot(exchange, intervals)
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 33, in __init__
    self.execute_cycle(interval)
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 72, in execute_cycle
    self.gather_data()
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 78, in gather_data
    self.get_historic_symbol_data()
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 148, in get_historic_symbol_data
    self.start_time_uri + str(start_date)).json()]
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with ur
l: /api/v1/klines?symbol=ETHBTC&interval=5m&limit=1000&startTime=1499040000000 (Caused by NewConnectionError('<urllib3.c
onnection.VerifiedHTTPSConnection object at 0x000001DCEC652BE0>: Failed to establish a new connection: [WinError 10060]
A connection attempt failed because the connected party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond'))
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
I've encountered similar but am struggling to remember the exact details of what happened.

I found this which may be useful:


tldr; is install pyopenssl, but whether or not the issue is completely related to what you're seeing I'm not sure
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
I've encountered similar but am struggling to remember the exact details of what happened.

I found this which may be useful:


tldr; is install pyopenssl, but whether or not the issue is completely related to what you're seeing I'm not sure
Thank you, that seemed to fix the issue(re-installing pyopenssl). It has been running now for 10+ hours, but I do not see it populating anything in mongodb.

(Windows) MongoDB version 3.2

I created the 'binance_historic' db with 'data5m' collection. Gave user 'binance_scrape' dbAdmin role for it. Did I miss anything?
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
Thank you, that seemed to fix the issue(re-installing pyopenssl). It has been running now for 10+ hours, but I do not see it populating anything in mongodb.

(Windows) MongoDB version 3.2

I created the 'binance_historic' db with 'data5m' collection. Gave user 'binance_scrape' dbAdmin role for it. Did I miss anything?

Good news. I don't think that was a fix I used previously and not sure exactly why it worked, but good that it did.

Regarding the time it takes, it took a LONG time for me as well, mainly because of the fact that it's building the indicators for so much raw data.

From memory, you should see notifications in the console regarding which stage it's currently at. If it's building indicators right now then it's fair that it would be running on a single phase for quite a while. The last time I used it the total processing time was over 15 hours I think, so it does take a while but the it's scraping all data from Binance since day-dot until now and building all of the historic indicators fresh from that data.
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
After a 12 hour run different errors popped up.

Code:
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\util\retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\packages\six.py", line 685, in reraise
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Program Files\Python37\lib\ssl.py", line 412, in wrap_socket
    session=session
  File "C:\Program Files\Python37\lib\ssl.py", line 853, in _create
    self.do_handshake()
  File "C:\Program Files\Python37\lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 473, in <module>
    data_bot = Data_Bot(exchange, intervals)
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 33, in __init__
    self.execute_cycle(interval)
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 72, in execute_cycle
    self.gather_data()
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 78, in gather_data
    self.get_historic_symbol_data()
  File "C:\Program Files\Python37\Scripts\data_bot_v3.py", line 148, in get_historic_symbol_data
    self.start_time_uri + str(start_date)).json()]
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\natha\AppData\Roaming\Python\Python37\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
I reinstalled both requests and urllib3. Hopefully that resolves the issue. I understand it can take 15+ hours, but I am curious why the mongodb is not getting populated. Is this expected?
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
The same two things are actually related. Everything is stored in memory, from the scrape through to the indicators being constructed.. so if the script fails before the indicators are built then the data hasn't been stored on the hard drive. The reason is that the scrape of the data itself actually doesn't take that long, it's building the indicators that takes a long time.

I haven't seen that error before, but would suggest Googling it (self._sslobj.do_handshake()urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(0, 'Error'))) and seeing if anything is relevant.

It's tough to say what it would be, but asking on Stackoverflow might help as well.
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
I reinstalled both requests and urllib3. Hopefully that resolves the issue. I understand it can take 15+ hours, but I am curious why the mongodb is not getting populated. Is this expected?
How did you end up going with that - Any luck solving it or still issues? I'll take a deeper look at the code if you're still struggling with it
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
reinstalling did not work either. I decided to run it on Python 3.6 instead of 3.7 and it has been going for 12+ hours now. Sitting at this spot now for an hour. Not sure if I should restart the process or just let it be.

Code:
 {'open_time': 1500304200000, 'open': 0.085604, 'high': 0.086258, 'low': 0.085538, 'close': 0.086165, 'volume': 20.227, 'close_time': 1500304499999}, {'open_time': 1500304500000, 'open': 0.086366, 'high': 0.086366, 'low': 0.085372, 'close': 0.085372, 'volume': 18.517, 'close_time': 1500304799999}]
1499040000000
 

BallsOhBalls

Neophyte
Joined
Mar 27, 2019
Messages
6
Reaction score
1
Location
USA
Found out recently that Gekko UI has a scrape tool built in. Using that instead since I am using it for trading.
 
Reactions: Entelechy

J.Babe.Abel

Neophyte
Joined
Dec 13, 2018
Messages
9
Reaction score
4
Sure we'll be releasing more scripts over the coming few days. They include:

- A script to gather a daily list of all new ICO's listed on the major platforms (goes through tens of thousands of listings to find the ones that have just been listed)

- A script to visualize the data gathered from the Binance API

- Considering also releasing a full script for a crypto backtesting engine as well.

Keep an eye on your alerts here so that you see when those scripts are released as well, or you can accept our push notifications also for updates. Thanks for your support!
Awesome!
 

Entelechy

Stadivm Owner
Staff member
Joined
Jul 8, 2018
Messages
161
Reaction score
30
Location
Stadivm.com
Hi,
very Nice project.
Can you add CCI and StochcasticRSI Please.

Thank you.
At this point there aren't any plans to additional indicators to this code, but I hope that if someone else has the ability to modify and a extra indicators they would repost here as well to help out others.