Use official ynab api and drop selenium webdriver #9

gitviola · 2018-02-21T22:03:23Z

Closes #10

Drop Figo Support (FinTS can be used for those banks)
Cleanup old selenium webdriver code
Support FinTS
Support N26
Support BBVA
Send transactions to ynab
Adjust readme (new docker service name, how to get access_token ect.)
Use official ruby lib https://github.com/ynab/ynab-sdk-ruby

wizonesolutions · 2018-06-02T13:16:12Z

Do I have to change my config at all to try this version? I guess I have to add my API key somewhere. I got early access.

gitviola · 2018-06-02T14:48:07Z

@wizonesolutions Hey, I already updated that part of the readme. Check out the config part. Basically it’s just adding the api key instead of your username and password: https://github.com/schurig/ynab-bank-importer/blob/5576c86846e5a894a10a6ab814b3779cc51894a4/README.md

Let me know how it goes. It works fine on my raspberry pi (without docker currently) - runs every 15 minutes on there.

When you test it out please create an extra budget first. If it works as expected then you can use yours. Note that all past transactions from the last 30 days (or 60 I think) might be imported the first time (the api offers a way to not import duplicates but the first time it doesn’t know it obviously). Once imported you can select all not approved ones and delete them - after that they shouldn’t be imported anymore.

Each transaction basically gets a hash calculated by it’s date, amount, payee name and some other stuff depending on the dumper. Even if you delete the transaction in YNAB, the next time the script uses the same calculated hash for a transaction the api won’t create it. Really cool way how they handle it. Only bank I know that changes transaction data sometimes so the hash changes is N26

wizonesolutions · 2018-06-07T12:39:30Z

Oh, noticed I had to switch from ruby to importer in my Docker command. I think that's already on your checklist. But I got this running docker-compose pull importer && docker-compose run importer:

 Default  ~/d/ynab-bank-importer   feature/ynab-api  docker-compose pull importer; and docker-compose run importer;
Pulling importer ... done
/usr/app/lib/ynab/user.rb:10:in `fetch': key not found: "username" (KeyError)
	from /usr/app/lib/ynab/user.rb:10:in `initialize'
	from /usr/app/run.rb:8:in `new'
	from /usr/app/run.rb:8:in `<main>'

I thought username was no longer needed now that access_token is used.

wizonesolutions · 2018-06-07T12:42:24Z

For the record, I tried removing the old running container, just in case, but that didn't help.

wizonesolutions · 2018-06-07T12:48:03Z

Oh. The image that it uses contains the old script. Should it be the beta tag or something? I will just build from the Dockerfile as schurig/ynab-bank-importer and see if that helps.

(I ran docker-compose run importer /bin/bash and then inspected lib/yaml/user.rb)

gitviola · 2018-06-07T13:28:00Z

Ah yes sorry, the correct image is tagged with beta. But you could also build it yourself. beta-rpi might also work if you work on a raspberry pi but it hold old code since the docker ci can't build those kind of images, so I had to build it on my raspberry pi and push it. Didn't do that for a long time.

So inside the docker-compose.yml file instead of schurig/ynab-bank-importer:latest put in schurig/ynab-bank-importer:beta. I didn't change the file because at some point latest will get the new code and maybe the old tag will move to selenium.

wizonesolutions · 2018-06-07T14:35:09Z

Seems to have worked now! I will try it with my real budget a little later. The YNAB API is convenient :) I wrote a tool to export transactions from a single account within a date range to the Xero CSV format.

wizonesolutions

Works well so far! The import_id is doing what it should, and transactions only get imported once. For N26, does it only import cleared transactions? That is important for e.g. foreign currency purchases. Since it uses the N26 API, I assume so, but was wondering about this.

Fidor's API has a Ruby client too, so I'm going to look into that at some point. It requires an access token/OAuth, so I don't really know how that flow will look.

wizonesolutions · 2018-06-11T14:54:47Z

Hmm, transaction matching doesn't seem to be as smart via the API as via CSV? None of the transactions we put in beforehand match and show the chain icon when the transactions come in via the API...

wizonesolutions · 2018-06-23T13:41:02Z

Also, N26 seems to be pulling in transactions that are too new (same day). There must be a "cleared" property in its API, no? Perhaps the script should filter on that. Because it's importing them to YNAB as cleared (implying final), but sometimes that is wrong.

I'll take a look sometime, since I want to try to avoid it never matching also.

The API is public now btw

gitviola · 2018-06-28T19:44:47Z

@wizonesolutions sorry for the delay. I saw that they announced the public api 🎉. That means that I will merge this PR asap.

Also about N26, yes currently it doesn't check if the transaction was actually cleared or not. I will add this asap too. My idea is to have a configuration flag for this. By default it will then be "only cleared" and you can set it to false. I like that I see those transactions right after my purchase because I have a hard time remembering what it was after a few days. But yes, if it's an other currency it can lead to problems ect. So I will add this.

Right now I don't have so much time because I'm busy with some other more important things. But I will try to do this as soon as I have time. In case you want to give it a try, feel free to do so.

I'm really glad to see people like you being interested in it! Thanks a lot for the comments 😊!

wizonesolutions · 2018-06-28T19:53:50Z

Btw the transaction matching thing is an API issue, and YNAB is working on it. It's apparently possible to select the two transactions and match them manually. I need to try it.

wizonesolutions · 2018-07-08T17:31:32Z

@schurig Hmm, this is strange. I'm looking at https://github.com/DanToml/twentysix/blob/master/lib/twentysix/core.rb, and I see that #transactions has include_pending set to false by default. I wonder why it is pulling them in anyway.

I searched your codebase, but I don't see you setting it to true anywhere. I don't think it's lock file issues, since that library hasn't been updated since early last year. I guess I'll try adding a configuration option, as you suggest, and filtering them on the tail end (by checking the pending property on the response). Maybe there is a bug in the library.

wizonesolutions · 2018-07-08T18:29:41Z

...ugh. Seems like N26's idea of pending is not the same as ours. Do you know by chance which property is changing that makes a new import_id be generated? Is it transactionNature?

(I logged in and checked my statement vs. what the API was saying...one of the ones that "isn't pending" doesn't show up on the statement yet, so I don't know how it determines this.)

gitviola · 2018-07-09T12:21:39Z

@wizonesolutions I think on their side they create a brand new transaction in their system and delete the old one. The id of the transaction always changes then some of the attributes change which is very sad. But I think they returned two different dates. One is the created one and the other one the effective one. And maybe we should here talk the one that is more in the future and filter all transactions out where that date is in the future

wizonesolutions · 2018-07-09T12:24:44Z

@schurig ...ouch. I wish there a way to limit transactions to ones in the statement. Maybe there's another API call or something. N26 itself on the web must be getting the statement transactions somehow, after all (I assume it uses its own API).

gitviola · 2018-07-09T12:44:43Z

@wizonesolutions yes, actually the API is their own private one (that they use for their mobile app). They don't have an official public one. But filtering them by a field on our side would be fine I think. Should be a 1-liner

https://github.com/DanToml/twentysix

wizonesolutions · 2018-07-09T13:26:40Z

@schurig Oh yes, but I don't think there is a field that indicates if a transaction is actually finalized or not. I don't know how they tell!

…

On Mon, Jul 9, 2018, 14:44 Martin Schurig ***@***.***> wrote: @wizonesolutions <https://github.com/wizonesolutions> yes, actually the API is their own private one (that they use for their mobile app). They don't have an official public one. But filtering them by a field on our side would be fine I think. Should be a 1-liner https://github.com/DanToml/twentysix — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAVANO7CRDqyJTzxN_CQQaAvNc3acn63ks5uE1A8gaJpZM4SOZ-U> .

* add known problems section * Explain how the API works * Explain how to optain an personal access token

gitviola · 2018-07-21T12:37:52Z

@wizonesolutions I now reject all pending transactions from N26 by default. So only the ones that are not pending anymore will be imported. Let me know how it works. Maybe in the future I will add a feature flag for n26 so you can set it how you want it.

wizonesolutions · 2018-07-26T10:35:34Z

@schurig what I experienced was that N26's pending flag was kind of misleading...transactions still import several times even with that. But I will try it again. I currently had it turned off and was using the CSV....but even the CSV contains transactions not in the statement! They are a strange bank...maybe we have to the transactions in YNAB as well to figure out what's going on...but that obviously adds complexity and potentially the need for a data store, etc. 😕

gitviola · 2018-07-26T12:15:43Z

@wizonesolutions yes I agree, that would add complexity if we would need to match transactions on our side. I like how stateless the script currently is..

Btw, if it imported transactions with the flag pending before, it should now ignore those because in the code I now explicitly check for the value and reject all transactions where this is true:

https://github.com/schurig/ynab-bank-importer/blob/b45418ebeb80fb67ffed8f0d5012b8092ac3e454/lib/dumper/n26.rb#L32-L34

wizonesolutions · 2018-07-26T12:24:19Z

@schurig Oh I saw that. What I mean is that I think most of the time N26 doesn't use the pending flag to mean "not finalized and on statement"...a.k.a. it's almost always false. And it can be false but the transaction still invisible on the current statement. There was a card verification charge from Amazon or something that never appeared on the statement, but it came in even when I had my own version of ignoring pending transactions in place.

gitviola · 2018-07-26T12:42:49Z

Oh I see.. That's so weird. Maybe they're re-structuring their api. At the end the API is not public. But there are good chances that they introduce less and less changes because they now also introduced a web-version that behaves similar to the mobile app, which means they now have to maintain 3 different things (ios, android and webapp) 😁

Use official ynab api

5e722e8

gitviola added the feature label Feb 21, 2018

Martin Schurig added 2 commits February 23, 2018 22:47

Add BBVA support, use ynab gem

058a30c

Detect BBVA cash withdrawal (closes #1)

101552e

gitviola mentioned this pull request Feb 23, 2018

[FEATURE] Detect ATM withdrawals #1

Closed

6 tasks

gitviola force-pushed the feature/ynab-api branch 2 times, most recently from c6599bd to eee1e7b Compare February 24, 2018 01:47

Add specs

8e59e5e

gitviola force-pushed the feature/ynab-api branch from eee1e7b to 8e59e5e Compare February 24, 2018 02:00

Martin Schurig added 2 commits February 25, 2018 00:17

Only strip existing strings

847f0e0

Add rubocop + codeclimate

e4b0751

gitviola force-pushed the feature/ynab-api branch 5 times, most recently from ed1a4c4 to 70e976f Compare February 25, 2018 00:42

Make rubocop happy

9517d03

gitviola force-pushed the feature/ynab-api branch from 70e976f to 9517d03 Compare February 25, 2018 19:57

Martin Schurig added 3 commits March 9, 2018 18:16

Don't import future transactions (api error)

e64c340

Fix bug where number was positive when it should be negative

5ec321d

Update to ynab 0.3.0

e10f0a0

gitviola force-pushed the feature/ynab-api branch from b075be4 to e10f0a0 Compare March 9, 2018 17:48

Improve import_id calculation for N26 and FinTS

c58553e

gitviola force-pushed the feature/ynab-api branch from 1c67d08 to c58553e Compare April 2, 2018 12:05

Cleanup rpi dockerfile

5576c86

philipp-paland mentioned this pull request May 28, 2018

YNAB has an early-access API #10

Closed

wizonesolutions reviewed Jun 8, 2018

View reviewed changes

Martin Schurig added 4 commits July 21, 2018 13:53

Improve README

79d3c1f

* add known problems section * Explain how the API works * Explain how to optain an personal access token

Only take N26 transactions that aren't pending anymore

8a6545c

Use official YNAB gem

fe6969a

Make rubocop happy

b45418e

gitviola merged commit 3677c8b into master Jul 21, 2018

gitviola deleted the feature/ynab-api branch July 21, 2018 12:38

gitviola mentioned this pull request Jul 23, 2018

Update README.md #11

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use official ynab api and drop selenium webdriver #9

Use official ynab api and drop selenium webdriver #9

gitviola commented Feb 21, 2018 •

edited

Loading

wizonesolutions commented Jun 2, 2018

gitviola commented Jun 2, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018 •

edited

Loading

gitviola commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions left a comment

wizonesolutions commented Jun 11, 2018

wizonesolutions commented Jun 23, 2018

gitviola commented Jun 28, 2018

wizonesolutions commented Jun 28, 2018

wizonesolutions commented Jul 8, 2018

wizonesolutions commented Jul 8, 2018 •

edited

Loading

gitviola commented Jul 9, 2018

wizonesolutions commented Jul 9, 2018

gitviola commented Jul 9, 2018

wizonesolutions commented Jul 9, 2018 via email

gitviola commented Jul 21, 2018

wizonesolutions commented Jul 26, 2018

gitviola commented Jul 26, 2018

wizonesolutions commented Jul 26, 2018

gitviola commented Jul 26, 2018

Use official ynab api and drop selenium webdriver #9

Use official ynab api and drop selenium webdriver #9

Conversation

gitviola commented Feb 21, 2018 • edited Loading

wizonesolutions commented Jun 2, 2018

gitviola commented Jun 2, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018 • edited Loading

gitviola commented Jun 7, 2018

wizonesolutions commented Jun 7, 2018

wizonesolutions left a comment

Choose a reason for hiding this comment

wizonesolutions commented Jun 11, 2018

wizonesolutions commented Jun 23, 2018

gitviola commented Jun 28, 2018

wizonesolutions commented Jun 28, 2018

wizonesolutions commented Jul 8, 2018

wizonesolutions commented Jul 8, 2018 • edited Loading

gitviola commented Jul 9, 2018

wizonesolutions commented Jul 9, 2018

gitviola commented Jul 9, 2018

wizonesolutions commented Jul 9, 2018 via email

gitviola commented Jul 21, 2018

wizonesolutions commented Jul 26, 2018

gitviola commented Jul 26, 2018

wizonesolutions commented Jul 26, 2018

gitviola commented Jul 26, 2018

gitviola commented Feb 21, 2018 •

edited

Loading

wizonesolutions commented Jun 7, 2018 •

edited

Loading

wizonesolutions commented Jul 8, 2018 •

edited

Loading