FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[schmidsi]

Hi

I wanted to try out this promising tool on our code base, but it fails with the following error:

$ yarn solidity-cli -i src/contracts/exchanges/EthfinexAdapter.sol -o out
yarn run v1.12.3
$ /Users/schmidsi/Development/@melonproject/protocol/node_modules/.bin/solidity-cli -i src/contracts/exchanges/EthfinexAdapter.sol -o out

<--- Last few GCs --->

[2974:0x104000000]    49824 ms: Scavenge 1144.6 (1439.2) -> 1141.2 (1440.2) MB, 13.6 / 0.2 ms  (average mu = 0.118, current mu = 0.091) allocation failure 
[2974:0x104000000]    50730 ms: Mark-sweep 1150.2 (1440.2) -> 1141.2 (1438.7) MB, 822.4 / 0.8 ms  (average mu = 0.145, current mu = 0.172) allocation failure scavenge might not succeed
[2974:0x104000000]    50875 ms: Scavenge 1153.0 (1438.7) -> 1149.3 (1441.2) MB, 13.2 / 0.2 ms  (average mu = 0.145, current mu = 0.172) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

   0: ExitFrame [pc: 0x2781412e37e1]
Security context: 0x2b15b2a1e549 <JSObject>
   1: stringSlice(aka stringSlice) [0x2b1515a2afe1] [buffer.js:~590] [pc=0x2781412ea5d5](this=0x2b15d56826f1 <undefined>,buf=0x2b15e4712e29 <Uint8Array map = 0x2b15ee24f729>,encoding=0x2b15b2a3c5c1 <String[5]: utf-8>,start=0,end=14622)
   2: oncomplete(aka readFileAfterClose) [0x2b15d22a4c71] [internal/fs/read_file_context.js:~31] [pc=0x2781412e6dd1](this=...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x100038c65 node::Abort() [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
2: 0x100038e41 node::OnFatalError(char const*, char const*) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
3: 0x1001b43a5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
4: 0x10057ee02 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
5: 0x1005818d5 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
6: 0x10057d77f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
7: 0x10057b954 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
8: 0x1005881cc v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
9: 0x10058824f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
10: 0x100559bb5 v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
11: 0x100559dd1 v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
12: 0x1001d6188 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
13: 0x1000c2223 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
14: 0x100050593 void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node]
15: 0x2781412e37e1 
16: 0x2781412ea5d5 
17: 0x2781412e6dd1 
18: 0x27814128ee55 
/var/folders/mc/skskq1wd615g5fkh7zrlt6nw0000gn/T/yarn--1543568322766-0.3745581330585297/node: line 3:  2974 Abort trap: 6           "/Users/schmidsi/.nvm/versions/node/v10.9.0/bin/node" "$@"

If you want to try to reproduce this on your machine, do the following steps (assuming solidity-cli is installed globally):

git clone https://github.com/melonproject/protocol.git
cd protocol
git checkout develop
solidity-cli -i src/contracts/exchanges/EthfinexAdapter.sol -o out

If you are interested, you can have a look at our current implementation of compiling: https://github.com/melonproject/protocol/blob/develop/bin/compile.ts

It works, but it is slow :(. That's why I'm here. (enzymefinance/protocol#581)

[pubkey]

It looks like the nodejs-memory is to low for your contract, maybe because of the amount of imports.
Can you increase the memory-size and try again?

[pubkey]

Maybe you have an infinite cycle of imports.
I think we need something to reproduce this in a simple way, not with 50 contracts importing each other.

[pubkey]

I could reproduce this by just making a circular import.
@schmidsi do you know if circular imports can be used with solc?

[schmidsi]

Probably. We used to use dapp tools on this set of contracts for compiling. Because it was so complicated to setup a dev-env, I started to create this JS only solution. Actually, my implementation works, but it is just really slow: https://github.com/melonproject/protocol/blob/develop/bin/compile.ts#L99-L158

So how is this caching working?

[pubkey]

I tried to solve the circular imports, but it's not as easy as i thought.
Let's wait until this is resolved and then we can switch to resolver-engine for the parsing.

[pubkey]

I looked into resolver-engine and it is not useable at the moment.
A better approach is to use the remix-parser, when it got splitted out from the project.
Related issue

[nventuro]

Is there an estimated timeline for this? I'd love to be able to setup compilation for multiple Solidity versions in OpenZeppelin without having to upgrade to truffle 5 (which would also mean updating the whole test suite due to web3 1.0 changes), and this project seems perfect for that.

Thanks!

[pubkey]

Hi @nventuro
Sorry but there is no timeline for that on this project. Than plan is to wait for resolver-engine and remix-parser to be useable and then switch to one of them.