Need a way to set content-filtering rules across the project

Bug #395731 reported by Frits Jalvingh
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
High
Unassigned
Breezy
Triaged
High
Unassigned

Bug Description

We need a way to set content filtering rules in the tree.

----

Context:
I want to use content filtering to fix the line endings horror. Content filtering only works when a lot of preconditions are met, otherwise it fails silently (simply does not work WITHOUT reporting an error) even though it is configured in .bazaar/rules.

[This by itself is a serious bug in my eyes because it causes silent corruption of the repository (files containing a full delta because their line endings are incorrect). This is quite horrible and cannot really be seen by the developer - he just sees the files he's worked on being committed. Only a long time after this merge will the problem be seen, when another merge is attempted causing a full conflict.]

Problem:
One of the preconditions to have content filtering work is to have "Working Tree Format 5", earlier versions do not support it (and do not fail but continue to work, committing garbage = files with incorrect line endings).
When you branch locally from a repository containing a "Working Tree Format 5" tree the resulting branch will be correct, and will have a version 5 tree also. This is correct.

However when branching from a smart server (which uses branches in a shared init-repo) you get the "default" working tree format 4, which is incapable of content filtering. This results in a working tree having incorrect line endings and corruption when files get commited having incorrect line endings.

Solution:
A proper solution would be to use at least format 5 when the repository format itself is capable of content filtering. Or to be able to specify a minimal format in the server repository.

For now I have to resort to patching bazaar for all my developers, changing the default working tree format to 5 in bzrlib/workingtree.py, last lines 8-(.

A final note:
In addition, it starts to become very important to be able to force some bazaar configuration into the tree somehow. The current solution for line endings depends on every developer having a properly configured workstation. Any failure to properly configure will cause problems in the repository which is shared by everyone, this is quite unacceptable. This does not ony go for line endings but also for things like required plugins (commit checkers and such).

It might be sufficient to have a version controlled ".bazaar-config" directory in the root of a tree containing the master rule file, and a list of required plugins and their version (not the plugins themselves). At this moment it is way, way easier to get things wrong than to get things right.

Revision history for this message
Martin Pool (mbp) wrote :

I agree, this is quite a problem.

The issue of wt format is bug 414305.

So I'm going to dedicate this bug just to the need for versioned rules.

description: updated
Changed in bzr:
status: New → Confirmed
importance: Undecided → High
summary: - Default working tree format INCORRECT for content filtered branches made
- from smart server
+ Need a way to set content-filtering rules across the project
Martin Pool (mbp)
tags: added: content-filtering
Revision history for this message
Craig Hewetson (craighewetson-deactivatedaccount) wrote :

This is a major problem for our team.
How do I check the working tree format version?
and how do i "upgrade" it to format 5?

I'm hoping that this will allow the content filtering to work then.
Because at the moment we get entire file changes that weren't supposed to be.

Revision history for this message
Alexander Belchenko (bialix) wrote : Re: [Bug 395731] Re: Need a way to set content-filtering rules across the project

Craig Hewetson пишет:
> This is a major problem for our team.
> How do I check the working tree format version?

`bzr info -v` will tell you the format of working tree.

> and how do i "upgrade" it to format 5?

only by upgrading entire branch+tree to --1.14 format:
`bzr upgrade --1.14`

Revision history for this message
Marius Kruger (amanica) wrote : Re: [Bug 395731] Re: Need a way to set content-filtering rules across the project

On 23 September 2010 09:15, Craig Hewetson <email address hidden> wrote:
> Context:
> I want to use content filtering to fix the line endings horror. Content filtering only works when a lot of preconditions are met, otherwise it fails silently (simply does not work WITHOUT reporting an error) even though it is configured in .bazaar/rules.
>
> [This by itself is a serious bug in my eyes because it causes silent corruption of the repository (files containing a full delta because their line endings are incorrect). This is quite horrible and cannot really be seen by the developer - he just sees the files he's worked on being committed. Only a long time after this merge will the problem be seen, when another merge is attempted causing a full conflict.]

I think the best way to guard against this sort of thing is to have a
check run on the central server before commit. i.e. install a
plugin[1] on the server that prevents these changes from occurring.
But then you can only allow smartserver access eg. bzr+https://

[1] I wrote a plugin that can do similar checks, if you like I can
extend it to be able to check line endings:
https://launchpad.net/bzr-text-checker

Revision history for this message
Frits Jalvingh (fjalvingh) wrote :

I just spent a happy three hours fixing our main repository *again*. A developer in a hurry has committed and merged code without proper rules configuration. This caused a commit which helpfully changed all 16000 files in the repository. I was lucky in that there were not too many commits and merges after it or I would have been forced to re-create our repositories from scratch *again*, losing all history in the process. Eye, needle, things like that.

In addition, I look forward to a nice Monday morning where I can help every developer to get things right on their local workstations.

I get a bit frustrated that it is this easy to destroy a main repository.

Is there any way to get this thing fixed faster? An easy way to handle this would be to just accept a "rules" file in the branch root somewhere (".bzrrules"). Would a patch doing that be accepted?

Revision history for this message
Martin Pool (mbp) wrote :

On 6 November 2010 22:25:21 UTC+11, Frits Jalvingh <email address hidden> wrote:
> Is there any way to get this thing fixed faster? An easy way to handle
> this would be to just accept a "rules" file in the branch root somewhere
> (".bzrrules"). Would a patch doing that be accepted?

Yes, a patch to do that would be very gratefully welcomed. That's
what Ian was working towards before he fell ill. If you want any help
with implementing or testing it, just ask.

--
Martin

Revision history for this message
Frits Jalvingh (fjalvingh) wrote :

Thanks; I'll go and work on it asap - but it will take some time.
I know Ian was working on it. He'll be missed an awful lot 8-(

Revision history for this message
John A Meinel (jameinel) wrote :

I believe the path we were aiming for was .bzr-meta/rules Just to avoid putting too many ".bzr" files in the root directory.

Revision history for this message
Frits Jalvingh (fjalvingh) wrote :

Thanks. I am thinking about what/how to handle it. I am writing a design doc at http://wiki.bazaar.canonical.com/fjalvingh, it might benefit from discussion when done - there are interesting issues with this..

Revision history for this message
Alexander Belchenko (bialix) wrote : Re: [Bug 395731] Re: Need a way to set content-filtering rules across the project

Frits Jalvingh пишет:
> Thanks. I am thinking about what/how to handle it. I am writing a design
> doc at http://wiki.bazaar.canonical.com/fjalvingh, it might benefit from
> discussion when done - there are interesting issues with this..
>
@Frits, I'm sure high-level directory should be .bzrmeta not
.bzr-meta. Because .bzrmeta is already used in bzr-externals and
scmproj plugins. I think .bzrmeta is already de-facto standard.
Please, don't create a new zoo of meta dirs.

--
All the dude wanted was his rug back

Revision history for this message
Marius Kruger (amanica) wrote : Re: [Bug 395731] Re: Need a way to set content-filtering rules across the project

I talked to Jelmer about this at the bzr-sprint at uds-m.
He would really like this sort of thing to *not* live in the workingtree,
because it makes it quite difficult to rountrip it to foreign vcss.
So the idea was to put it in .bzr/branch/rules but then it needs to
get propegated like .bzr/branch/tags - which is more work.
I was looking at this a while ago but got sidetracked.

Revision history for this message
Frits Jalvingh (fjalvingh) wrote :

Thanks for the extra info, gents. Having no wish to become a zookeeper ;-) I will use whatever name will become the "standard" if it is decided this is the way to go.
Marius, I wonder how you would maintain that file during merges et al? If branch a is merged into branch b, how is this file maintained? Has there been talk about workspace file handling that occur due to changes to this file?

Revision history for this message
Marius Kruger (amanica) wrote :

On 10 November 2010 16:20, Frits Jalvingh <email address hidden> wrote:
> Marius, I wonder how you would maintain that file during merges et al? If branch a is merged into branch b, how is this file maintained? Has there been talk about workspace file handling that occur due to changes to this file?

I was wondering about that too. If its in the workingtree then it
would get merged like any other text file. With tags I think we just
overwrite the target values.
I suppose we should be able to use our normal merging algorithms if it
is in .bzr/branch/rules too, but then we need to tell the user about
conflicts in it etc.

Its becoming a second versioned dir. This is sort of what people have
been asking for versioned properties. We can have a nested tree inside
eg. .bzr/branch/versioned-properties which can contain everything we
want (.bzr ignore rules conf andotherstuff)... its so crazy it just
might work (just need to make sure we dont recurse)

Revision history for this message
Frits Jalvingh (fjalvingh) wrote :

Ok, my first proposal was delightfully naive 8-(. Always fun to see something that appears to be easy to have a huge pile of hair..
So, I did some more thinking to try to get a more failsafe solution- and it proves to be quite nasty. I have written down the details as far as I see them now at http://wiki.bazaar.canonical.com/fjalvingh. A short rundown of the conclusions is:
* Any filtering rule change MUST go through the bzr UI. The file should NOT be user-editable at all (there is an alternative but it is more ugly)
* Changing rules is allowed for a non-dirty working tree only
* Changing rules may (must) cause the rewrite of the working tree, forcing those files to be proper for the new rules.
* Some changes must fully-change the repo with a commit. It would be good if the user would be warned when this happens because merging stuff from before that point could be almost impossible..

@Marius: thanks for your answer. I see some trouble with maintaining the rules file there and imposing a normal (file based) merge on it. I don't like the idea of my users mucking around in there- there's all kinds of stuff below .bzr and allowing them to edit there (as they will when there is a conflict) can open cans of worms attracting fishy problems.

The versioned properties thing looks more like it, although the merge rules for the rules file would actually be quite funny if you do it right; it should not just give two copies of a big multi-wildcard spec there. I would be surprised if actually merging files would be a good idea there. It might be best just to merge serialized structures using a special UI.

Vincent Ladeuil (vila)
tags: added: config
Vincent Ladeuil (vila)
tags: removed: config
Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
tags: added: views
removed: check-for-breezy
Changed in brz:
status: New → Triaged
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.