[~oli], thank you for your report. There already was an existing report MDEV-38671 about mostly the same, namely that the {{innodb_buffer_pool_size}} cannot be increased anymore when MariaDB Server has been started with the default settings.
The InnoDB deadlock detector was rewritten in MySQL 5.7.1, in a merge commit that does not mention any WL or bug number, nor any reviewer. I did not review those changes, and I cannot say whether the algorithmic complexity of the deadlock checker was reduced. Possibly the rewrite replaced procedural recursion with iteration over a data structure that explicitly represents a stack.
