Crystal projects don't properly install and build due to "pull" copying lib/ and bin/ folders from host

Bug #1936671 reported by Alexis Reigel
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Snapcraft
In Progress
Medium
George Dietrich

Bug Description

The initial error i got was:
`Failed to copy 'libevent-2.1.so.7': no such file or directory.`

This error occurs because the lib on the host system doesn't match the version on the snap's multipass instance. `shards install` doesn't update the lib/ and bin/ folder on the multipass instance, which leads to a stale build.

My current workaround is to delete the two folders on "pull" to ensure that the shards are built on the multipass instance:

```
override-pull: |
  snapcraftctl pull
  rm -rf $SNAPCRAFT_PART_SRC/lib $SNAPCRAFT_PART_SRC/bin
```

I'm not sure if this issue is specific to Crystal, or if the behaviour is fundamentally problematic, and built files from the host shouldn't be copied over at all. Potentially `.gitignore` could be considered.
If the issue is specific to Crystal we could add the delete command or something similar to https://github.com/snapcore/snapcraft/blob/master/snapcraft/plugins/v1/crystal.py#L92.

Tags: craft-414
Revision history for this message
Sergio Schvezov (sergiusens) wrote :

This is a hard to solve generically. We will ask the crystal team to be able to output to a specific directory so things don't conflict. In parallel, we will look into providing filters for pull and build just like we do for stage and prime.

Changed in snapcraft:
status: New → Triaged
tags: added: craft-414
Revision history for this message
George Dietrich (blacksmoke16) wrote :

I'm a bit confused on why deleting `lib` and `bin` from the Crystal project would resolve this. Those directories are git ignored by default when creating a Crystal project. They also have nothing to do with external .so or .a files as they represent dependencies of the shard itself and executables it built via `shards build`.

This seems to be more so related to extra ELF stuff not being being able to find the libs it needs to link.

Revision history for this message
George Dietrich (blacksmoke16) wrote (last edit ):

Okay, so I think I figured it out. I'm able to consistently reproduce running it locally on both the old and new plugins, and on 4.8.3 and 5.0.0 versions of snapcraft. @koffeinfrei some questions for you:

1. Are you also building this locally? Or via a build process or something?
2. Does your shard have any dependencies that put things within bin/? E.g. ameba

It seems that when these two cases are true: the `bin/` directory gets included within the VM. If this directory includes another binary (like ameba in my case) the ELF logic in the build method tries to find the libs for that binary (which was build on my local PC) whereas it works fine in the case of the actual file via `shard build` due to it actually being built within the VM.

However, if I were to `rm -rf bin/ lib/ shard.lock` before running `snapcraft` it works fine. Similarly it also works fine when setting the source to git URL, versus `./`. This is because these directores wouldn't exist, nor would they be installed due to the `--without-development` flag.

I suppose the easiest solution would be to just delete this directory within `pull` of the plugin to ensure there are no conflicts in this specific context. However I'm not super familiar with the plugin API, so I'll defer to @ergiusens to see if he has a better idea. Something more holistic could always be implemented in the future, but this would at least fix this issue in this context.

EDIT: After re-reading the initial bug report I realized the cause was already known, oops.

Revision history for this message
Alexis Reigel (koffeinfrei) wrote (last edit ):

> 1. Are you also building this locally? Or via a build process or something?

Yes, locally. If the host machine doesn't have a build (as would be the case with a CI env.) the problem doesn't occur.

> 2. Does your shard have any dependencies that put things within bin/? E.g. ameba

Exactly. The built binary is copied over from the host to the container, which doesn't have the same built environment and a different version of the linked library.

Changed in snapcraft:
assignee: nobody → George Dietrich (blacksmoke16)
Revision history for this message
George Dietrich (blacksmoke16) wrote :

I put together a PR for the quick fix to at least unblock this issue going forward: https://github.com/snapcore/snapcraft/pull/3567.

Changed in snapcraft:
status: Triaged → In Progress
Revision history for this message
Sergio Schvezov (sergiusens) wrote :

We are considering adding a source-ignore to be able to define specific paths.

Changed in snapcraft:
importance: Undecided → Medium
Revision history for this message
George Dietrich (blacksmoke16) wrote :

@sergiusens would that be baked into the plugin while also being configurable via the config file I assume? If we think that would be happening soon I can just close the workaround PR and re-create it in order to utilize that feature.

Revision history for this message
Sergio Schvezov (sergiusens) wrote :

The idea would be for parts follow either a .snapignore or .gitignore. Not decided yet, this is a similar pattern we followed for charmcraft.

Revision history for this message
George Dietrich (blacksmoke16) wrote :

@sergiusens That would be nice. Do you think that will be happening soon? Sounds like there are still some things to figure out, so could probably just move forward with that PR in the meantime?

Revision history for this message
George Dietrich (blacksmoke16) wrote :

@sergiusens Any word on the status of that new pattern and/or if https://github.com/snapcore/snapcraft/pull/3567 should be merged? Just ran into this again when testing a new build of `oq`.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.