[WIP] consider order .init_array .fini_array

[YamasouA]

Closes #588

[davidlattimore]

Great to see this progressing! Let me know if want any feedback yet or need any help

[YamasouA]

@davidlattimore
Thanks! A quick status update:

I’ve wired up the basic priority handling for .init_array / .fini_array and the legacy .ctors / .dtors, and all the unit tests are passing. I’m currently down to 6 failing integration tests, and in all of them the only remaining difference between wild and ld/lld is the order of entries in the final .init_array section (e.g. the ordering of frame_dummy, init_have_lse_atomics, ARGV_INIT_ARRAY, etc.).

Right now I’m trying to match GNU ld’s behavior more precisely here – especially how it derives init priorities from section names and how it handles legacy .ctors when converting them into .init_array (including the reversal of the ctor list). I have a rough plan, but I’m still figuring out the cleanest way to express this in wild’s codebase, so if you have any suggestions or pointers on how you’d structure this logic, I’d really appreciate your feedback.

[davidlattimore]

Can probably delete this TODO as well :)

[davidlattimore]

We need to sort all the .init_array sections over all the input files, but here, a new SortedSectionCollector is being created for each group. See the groups_and_buffers.into_par_iter() above. So at the moment, we'll only be sorting within a group. Which files go in which groups depends on things like the number of threads. You can force each file to be in a separate group for testing purposes by setting the environment variable WILD_FILES_PER_GROUP=1. You might find that an init-order test that passes at the moment, fails if run with this variable set.

So I think this line will need to go in the outer level of write_file_contents. That would mean that it'd need to be shareable between threads. It could either be wrapped in a Mutex or the collection could be changed to a concurrent collection like crossbeam_queue::SeqQueue. However, even that, I don't think will work because the buffers are per-group, so when you try to call sorted_sections.flush after the try_for_each has finished, you won't be able to because you don't have access to a buffer. The problem is that the buffers for .init_array etc have been allocated to each group. I think we really want one buffer for the whole .init_array section that's the right size for all of the contributions coming from different input files. Most likely that one buffer would need to be allocated to the epilogue.

One way this might be achieved is during layout, the groups from which the .init_array sections originate, rather than requesting space in the .init_array output section, could ask the epilogue or prelude to do so. One way to do this might be to add a new variant to layout::WorkItem. The epilogue when it receives a request to handle a sorted section can record information about that section, allocate space for it etc. In Epilogue::finalise_layout, we can then sort those sections by their priority. Writing the output section would then be done entirely by the epilogue.

[davidlattimore]

To add a little more detail and alternatives...

The new variant of WorkItem can probably be similar to WorkItem::LoadSection. It can probably even reuse SectionLoadRequest. e.g.`

LoadOrderedSection(SectionLoadRequest),

Such requests should only be sent to the epilogue, so if a non-epilogue group receives such a request, it's fine to panic.

An alternative that might be simpler would be to add an extra field to GraphResources. That field could hold for example an OutputSectionMap<crossbeam_queue::SegQueue<SectionToSort>> or something like that. Each input file, upon finding an input section that it wants sorted could add to that queue for the relevant output section. The epilogue, probably in finalise_sizes could take those queues and allocate whatever space is needed in each sorted output section. It could do the actual sorting either in finalise_sizes or finalise_layout.

[YamasouA]

@davidlattimore
Thank you for your advice.!!
I try to Implement GraphResources field.

[davidlattimore]

Hmm, this is tricky. I hadn't given much thought to symbols. The symbols are still owned by the object, it's just the section that is being written by the epilogue. I suspect we'd need the epilogue to determine the resolutions for the sections, then populate the relevant resolutions on the object so that the object can then resolve symbols that refer to that section. This is making me wonder if my suggestion to transfer ownership to the epilogue might have been a bad idea.

An alternative to transferring ownership would be to have multiple OutputSectionIds for .init_array, one for each used priority level. These would be merged together - see SectionKind::Secondary.

I've made a couple of refactorings that I hope would make that approach easier. It's possible that more refactorings still should be done. e.g. currently it might be a bit annoying to put __init_array_end in the correct place, although I think it should still be doable. It'd just need to be attached to the last OutputSectionId that merges into .init_array. Alternatively, we could add a second field similar to end_symbol_name, but which places the relevant symbol after any other secondary sections.

I'd be happy to go into more details if you'd like. I'm also happy to make more refactorings to make it easier.

Anyway, just know that there is an alternative if you decide that moving stuff to the epilogue is proving too difficult and sorry if I possibly sent you down the wrong path.

[davidlattimore]

I've only skimmed over the changes so far. One thing that I wanted to check is are the tests passing for you, but failing in CI? If so, make sure you have run_all_diffs = true in your test-config.toml.

[davidlattimore]

This line was removed in a separate PR. It's replaced by the timing_phase! line inside the function.