Commit Graph

695 Commits

Author SHA1 Message Date
Marco Boss
b4f7fef224
stay with partitions for now - find shards on dev/boss/sharding 2022-03-01 15:54:47 +01:00
Marco Boss
0eed0c0de2
tested shard creation locally 2022-03-01 14:01:42 +01:00
Marco Boss
afb1b5f902
prepare sharding / partitioning init
- create exchange-0001.sql with functions to create all tables and partitions / shards
- create exchange-0002.sql for the master db
2022-03-01 12:11:58 +01:00
Christian Grothoff
e24a6369aa
test-sync passes again 2022-02-28 23:18:40 +01:00
Christian Grothoff
a8d9e7e0a6
-more bool cleanup 2022-02-28 22:46:24 +01:00
Christian Grothoff
3d4e580699
work on auditor-sync update to latest DB 2022-02-28 22:41:24 +01:00
Christian Grothoff
ee79f31275
-work on test_sync.sh, incomplete 2022-02-28 22:41:24 +01:00
Marco Boss
0efc7fd524
cleaner sql 2022-02-28 22:32:59 +01:00
Christian Grothoff
23bc09fe3c
get test-auditor and #7181 to pass 2022-02-28 20:37:19 +01:00
Christian Grothoff
8f77bda2e0
-towards getting auditor tests to pass again with new logic 2022-02-28 16:13:30 +01:00
Marco Boss
58eb3d95fc
towards adding shard/partitioning init functionality 2022-02-27 21:23:20 +01:00
Marco Boss
99dd36f7f6
add index on wire_targets
- better performance on local db
- many queries use wire_target_serial_id as join condition -> discussion use this as shard key?
2022-02-26 17:37:52 +01:00
Marco Boss
3d52f52164
Fix ON CONFLICT statements for distributed tables 2022-02-24 14:31:19 +01:00
Christian Grothoff
10b9023575
-on conflict example 2022-02-24 10:16:10 +01:00
Christian Grothoff
5e5004020e
-fix aggregator with deferred constraint 2022-02-24 10:06:43 +01:00
Christian Grothoff
e253a5b055
-unique constraint is crucial 2022-02-24 09:06:15 +01:00
7910ca183f
double melt test no works with age restriction
- added missing field h_age_commitment in exchange's error response
- slight refactoring
2022-02-22 18:35:10 +01:00
26158fc725
[age restriction] progress 16/n - refresh/reveal/link tests
Age restriction works now with withdraw, melt/refresh/reveal and link,
including tests.

However, there is still a problem with the tests:  The melting operation
"refresh-melt-failing-age" that should fail (because of conflict), but
currently fails for other reasons.  I decided to disable that particular
test (and the next) and submit the patch I have so far.
2022-02-22 14:34:47 +01:00
Christian Grothoff
544ba42f44
-big renaming of structs for consistent naming with P suffix 2022-02-21 00:23:23 +01:00
Christian Grothoff
19624fd776
-ensure different HKDF is used in melt vs. withdraw 2022-02-20 21:51:46 +01:00
Christian Grothoff
bc150693de
-work around missing constraint, if not present 2022-02-20 19:26:01 +01:00
71af4c539e
-reducing FIXMEs re: age restriction
- deposit info now carries h_age_commitment
- benchmark does age commitment, too
2022-02-18 02:07:38 +01:00
f4f502d037
-minor merge conflict resolves 2022-02-18 00:50:31 +01:00
a78b3345fb
[age restriction] progress 15/n - melt/refresh/reveal and recoup
Added age restriction support for
  - melt/refresh/reveal
  - recoup

However, tests are not yet implemented for those flows.

Also: minor fixes and refactoring.
2022-02-18 00:30:19 +01:00
Christian Grothoff
22fe5da700
-migrate to Postgress 11: use BIGINT GENERATED BY DEFAULT AS IDENTITY instead of BIGSERIAL 2022-02-17 23:08:34 +01:00
Christian Grothoff
f74757e234
-another OEC FIXME 2022-02-17 15:28:00 +01:00
Christian Grothoff
721c2c88d5
-use age_mask 2022-02-17 15:23:56 +01:00
Christian Grothoff
a351bfc4b4
-fix CS nonce reuse check logic 2022-02-17 15:18:09 +01:00
8bdf6ab19d
[age restriction] progress 14/n - withdraw and deposit
Age restriction support for
  - withdraw is done and tested
  - deposit is done and tested

TODOs:
  - melt/refresh/reveal
  - link

------

Added functions
 - TALER_age_restriction_commit
 - TALER_age_commitment_derive
 - TALER_age_commitment_hash
 - TALER_age_restriction_commitment_free_inside
 - Hash of age commitment passed around API boundaries

Exchangedb adjustments for denominations
 - all prepared statements re: denominations now handle age_mask
 - signature parameters adjusted

Hash and signature verification of /keys adjusted
 - Hashes of (normal) denominations and age-restricted denominations are
   calculated seperately
 - The hash of the age-restricted ones will then be added to the other
   hash
 - The total hash is signed/verified

Tests for withdraw with age restriction added
 - TALER_EXCHANGE_DenomPublickey now carries age_mask
 - TALER_TESTING_cmd_withdraw_amount* takes age parameter
 - TALER_TESTING_find_pk takes boolean age_restricted
 - WithdrawState carries age_commitment and its hash
 - withdraw_run derives new age commitment, if applicable
 - Added age parameter to testing (13 as example)

Various Fixes and changes
 - Fixes of post handler for /management/extensions
 - Fixes for offline tool extensions signing
 - Slight refactoring of extensions
 - Age restriction extension simplified
   - config is now global to extension
   - added global TEH_age_restriction_enabled and TEH_age_mask in
     taler-exchange-httpd
   - helper functions and macros introduced
2022-02-16 22:01:05 +01:00
Christian Grothoff
ef938e0f7a
-correctly implement CS idempotency check on withdraw 2022-02-15 17:07:13 +01:00
Christian Grothoff
bd77bcb52d
-towards fixing the protocol 2022-02-14 23:02:25 +01:00
Christian Grothoff
c2549e8b1e
-address FIXMEs 2022-02-12 11:24:32 +01:00
Christian Grothoff
bc14c215b3
-doxygen fixes 2022-02-12 11:12:33 +01:00
Christian Grothoff
819b67426c
-doxygen fixes 2022-02-12 10:33:23 +01:00
Christian Grothoff
3b1e742dde
-simplify: nonce no longer hashed 2022-02-11 18:00:20 +01:00
Christian Grothoff
532d4ad0dc
-fixes to tests, and half-baked fixes for CS-/link (still fails) 2022-02-10 23:39:00 +01:00
Christian Grothoff
b2e6fcae1a
fix DB API for generic blinded planchet storage 2022-02-07 13:14:25 +01:00
Christian Grothoff
b9ea075d61
get DB test to build 2022-02-07 12:42:27 +01:00
Christian Grothoff
fb9ba5b1d2
-rename fest 2022-02-07 12:33:35 +01:00
Gian Demarmels
752c0aca43
uncrustify 2022-02-04 19:50:21 +01:00
Gian Demarmels
37f54d3e5d
exchangedb fix 2022-02-04 19:50:12 +01:00
Gian Demarmels
a67786078b
resolves merge conflicts 2022-02-04 16:50:32 +01:00
Gian Demarmels
086cf05794
refactor TALER_coin_ev_hash 2022-02-04 15:37:34 +01:00
Gian Demarmels
2213012866
include denom_pub into coin_ev_hash 2022-02-04 15:37:34 +01:00
Lucien Heuzeveldt
8d85c8b5b6
implement feedback 2022-02-04 15:37:30 +01:00
Gian Demarmels
5b7e8f9ac5
refactoring 2022-02-04 15:36:11 +01:00
Gian Demarmels
9c2aefaa51
removed varargs 2022-02-04 15:36:08 +01:00
Gian Demarmels
f3fb7c29e6
added CS data structures, implemented CS keypair 2022-02-04 15:31:45 +01:00
1962ed6b0b
improvements in extension handling
- extensions_sig is needed globally
- keep original json with config of extension
- fixed various bugs re: extension handling
2022-01-22 00:26:43 +01:00
0b56de6c99
[age restriction] progress 12/n
- taler-offline-tool now handles extensions
  - command "extensions" added with subcommands "show" and "sign"
  - parses extensions from taler config
  - shows and signs of extensions and their configurations
  - creates signed set of configurations for upload
  - added test for retrieval of extension config

- simplified signature verification for extensions
  - remove per-extension signatures, also from DB schema
  - adjust prepared statements accordingly
  - adjust DB event handler for extensions
  - allow NULL for config for extension in DB schema
  - handler for /management/extensions adjusted to new datastructures

- changed test for TALER_denom_blind/TALER_denom_sign_blinded with and
  without TALER_AgeHash

- minor updates and various fixes
2022-01-21 15:41:02 +01:00
Jonathan Buchanan
c10b783521
use 'pipe' instead of 'eventfd' on non-Linux systems 2022-01-18 09:15:54 -05:00
Christian Grothoff
766a291151
fix #7143 2022-01-11 17:56:32 +01:00
Christian Grothoff
e7aeec04f4
The current recoup API is broken. I guess this is another example where "trivial" API changes turn out to have (multiple!) unexpected consequences.
The current "/recoup" API does not have clear idempotency semantics, as we've discussed on the phone.  This is already bad by itself, as it makes it hard to write down what the API does other than "whatever the implementation does".

However, it actually breaks correctness in this (admittedly kinda contrived, but not impossible) case:

Say that we have a coin A obtained via withdrawal and a coin B obtained via refreshing coin A. Now the denominations of A gets revoked..

The wallet does a recoup of A for EUR:1.

Now the denomination of B also gets revoked.  The wallet recoups B (incidentally also for EUR:1) and now A can be recouped again for EUR:1.  But now the exchange is in a state where it will refuse a legitimate recoup request for A because the detection for an idempotent request kicks in.

This is IMHO bad API design, and the exchange should simply always recoup the maximum amount.

Furthermore, we usually follow the principle of "API calls that take up DB space are paid".  With the current recoup API, I can do many tiny recoup requests which the exchange then has to store, right?

I guess it would not be a big change to remove the "amount" value from the recoup/recoup-refresh request bodies, right?

- Florian
2022-01-11 12:47:35 +01:00
Christian Grothoff
aaaaa9a103
fix amount denormalization issue 2022-01-10 09:04:09 +01:00
d91750ca0f
drop extensions table 2022-01-08 19:45:19 +01:00
d6bf20028d
Fixed prepared statement for extensions 2022-01-08 19:45:00 +01:00
cc7d7707ab
[age restriction] progress 10/n
More work towards support for extensions:
- Prepared statements and DB-plugin-functions for setting and retrieving
  configurations from the database added.
- primitive "registry" of extensions for age restrictions and peer2peer
  (stub)
- TALER_Extensions now with FP for parsing, setting and converting a
  configuration.
- /management/extensions handler now verifies signature of the (opaque)
  json object for all extensions.
- /management/extensions handler calls the FP in the corrensponding
  TALER_Extension for parsing and setting the configuration of a
  particular extension

More work towards age restriction:
- TALER_Extensions interfaces for config-parser, -setter and converter
  implemented for age restriction
- DB event handler now retrieves config from database, parses it and
  sets it (the age mask) in the global extension.
- load_age_mask now loads age mask from the global extension (and not
  from the config file)
- add age_restricted_denoms to /keys response
2022-01-08 14:40:20 +01:00
Christian Grothoff
5cd2bc5de3
-cleanup 2021-12-28 15:36:26 +01:00
Christian Grothoff
070f442a11
resolve #7124: ensure -s also deletes work_shards 2021-12-26 13:05:39 +01:00
Christian Grothoff
e7d4183d1c
-fix FTBFS 2021-12-25 17:22:54 +01:00
Christian Grothoff
74e4c6429a
clean up refresh-reveal logic, reducing local copying 2021-12-25 16:13:06 +01:00
Christian Grothoff
f6ecb6c895
-eliminate redundant hash operation on link signatures 2021-12-25 15:39:01 +01:00
Christian Grothoff
87376e02eb
protocol v12 changes (/recoup split, signature changes) plus database sharding plus O(n^2)=>O(n) worst-case complexity reduction on coin balance checks 2021-12-25 13:56:40 +01:00
1b23857f2c
[age restriction] progress 8/n
More work towards support for extensions and age restriction

- updated gana
- added handler for DB-Event
- added TEH_extensions_init() and _done()
- added global for age restriction
- added stub for post handler
- added SQL-table for extension metadata
- added enum type for extensions and other data structures

Also:
- fixed some warnings -Wmaybe-unitialized
2021-12-23 15:15:33 +01:00
Christian Grothoff
62444504f7
denormalize reserve_pub in reserve tables to enable sharding 2021-12-19 13:43:33 +01:00
Christian Grothoff
1acc851deb
-fix recoup ugliness 2021-12-16 20:18:44 +01:00
Christian Grothoff
fdfcb29027
-fix FTBFS 2021-12-15 11:36:36 +01:00
Christian Grothoff
1a1fafbd43
introducing GNUNET_TIME_Timestamp, recoup now with amounts 2021-12-14 16:04:40 +01:00
Christian Grothoff
bf54ee30d4
experiment 2021-12-13 12:28:28 +01:00
Christian Grothoff
386301d1cf
-was integrated 2021-12-09 23:17:15 +01:00
Christian Grothoff
fba91c63d5
introduce stored procedure for coin balance check 2021-12-09 23:13:39 +01:00
Christian Grothoff
7fdcec4c3c
-complete 'melt.sql', in theory 2021-12-08 20:52:23 +01:00
Christian Grothoff
e0700ad916
optimize /deposit logic to minimize serialization failures (presumably) 2021-12-08 20:33:14 +01:00
Christian Grothoff
21951eacc2
quick hack to see about improving melt serialization failures 2021-12-08 18:22:07 +01:00
Christian Grothoff
9d3f7d3a56
move ensure_coin_known for melt and recoup outside of transaction, more logging 2021-12-08 15:18:40 +01:00
Christian Grothoff
a727348638
-wip 2021-12-08 11:53:27 +01:00
Christian Grothoff
7dcd217b60
-fix balance calculation 2021-12-06 16:37:32 +01:00
Christian Grothoff
f3629bdafc
improve SQL comments 2021-12-06 15:15:59 +01:00
Christian Grothoff
6ba9f934c4
-add FIXMEs 2021-12-05 19:23:16 +01:00
Christian Grothoff
19de0f604f
change order of statements when making coin known to possibly reduce serialization failures 2021-12-05 17:59:02 +01:00
Christian Grothoff
67de20d26e
major rework of withdraw transaction to use stored procedure and (presumably) reduce serialization failures by avoiding SELECT before INSERT 2021-12-05 17:16:00 +01:00
Christian Grothoff
c8480d97a1
try to optimize aggregator queries 2021-11-29 10:09:21 +01:00
Christian Grothoff
65533e8df5
-simple DB benchmark 2021-11-29 00:17:09 +01:00
Christian Grothoff
c3ce84af44
fix config 2021-11-28 22:20:56 +01:00
Christian Grothoff
cba49056c4
benchmark skeleton 2021-11-28 19:52:59 +01:00
Christian Grothoff
2d1a618d3d
sql fix 2021-11-27 22:58:50 +01:00
Christian Grothoff
e46fb30965
refactor get_ready_deposits query and index to hopefully make query faster 2021-11-27 14:56:28 +01:00
Christian Grothoff
ef29658161
fix #7107 2021-11-24 00:25:17 +01:00
Christian Grothoff
cb96bb0ba7
-fix warning 2021-11-21 15:16:58 +01:00
Thien-Thi Nguyen
2c8bd158d9
fix typo: s/row_id/rowid/ 2021-11-20 06:10:51 -05:00
Christian Grothoff
52fe9d657b
-fix leaks and compiler warnings 2021-11-19 20:56:53 +01:00
Christian Grothoff
326110ffe9
-improve comments, DCE 2021-11-19 16:45:55 +01:00
Christian Grothoff
6b8c757887
towards merging with master 2021-11-17 11:26:19 +01:00
Christian Grothoff
ed1db63a48
fix DB query and error handling 2021-11-17 11:01:19 +01:00
Christian Grothoff
422b0ffd4c
fix oauth_username issue 2021-11-16 19:31:27 +01:00
591cefd315
[age restriction] progress 3/n
- remove redundant struct members for age commitment hash
- undo pre-mature implementation in taler-exchange-secmod-rsa.c
- carry through age restriction parameter for TALER_denom_blind
2021-11-16 14:43:54 +01:00
Christian Grothoff
e5ead88057
complete oauth logic (in theory) 2021-11-15 20:00:45 +01:00
Christian Grothoff
0325a79631
avoid duplication 2021-11-15 14:39:18 +01:00
Christian Grothoff
016551dbb4
misc TODOs for KYC left 2021-11-09 20:36:30 +01:00
Christian Grothoff
98c30bee88
fix payto:// construction routine and move it to libtalerutil 2021-11-07 19:22:12 +01:00