Problem with \DeclareNameFormat in v. 3.3: number of arguments have changed.

[u-fischer]

The following \DeclareNameFormat definition (shorten from a longer version) works fine with older biblatex versions (v3.2) but fails with error in the newest version (v3.3).

! Illegal parameter number in definition of \blx@defformat@d.
<to be read again> 
                   3
l.12       \usebibmacro{name:last}{#1}{#3
                                         }{#5}{#7}%
? x

\documentclass{article}
\usepackage[style=authoryear]{biblatex}

\DeclareNameFormat{sortname}{%
      \usebibmacro{name:last}{#1}{#3}{#5}{#7}%
     }
\begin{document}


\end{document}

From the code and the documentation I can see that the cause is a quite dramatic change in the interface of \DeclareNameFormat. It now accepts only one argument. This adaption seems to work:

\documentclass{article}
\usepackage[style=authoryear]{biblatex}

\DeclareNameFormat{sortname}{%
    \nameparts{#1}
     \usebibmacro{name:family}
      {\namepartfamily}
      {\namepartgiven}
      {\namepartprefix}
      {\namepartsuffix}%
     }
\begin{document}

\end{document}

But how can one in this case write a suitable name format that works in the new and in older versions?

[plk]

I am honestly not sure. I tried several ways but the internals of name formats was a major problem stopping generalisation of name handling and so I had to chose an incompatible change. If you can devise something which will provide compatibility, I can put it into 3.4 but I think it's quite hard. For biblatex-apa, I just make a requirement for biblatex 3.3+ - I think this is reasonable given how easy upgrading is with TeXLive and MikTeX etc.

[u-fischer]

There are really a lot of people around which uses older system, and you can't write a biblatex style for e.g. a journal and require that all authors use the newest version. Is/was \abx@version always a (decimal) number so that one can safely test it? Then I will implement two versions depending on the value. (But I think that they should be some examples in the docu that show how to convert one definition in the new form).

[plk]

I realise it’s a bit painful but I honestly think there have to be jumps like this to get functionality in place for future expansion, otherwise, development of the system is just too slow. It’s also a matter of time - I have to put the major work into biblatex when I can and this means some releases have to be like this. I’m trying to get it to a state where things have been generalised to the point where this won’t happen again.

Anyway, \abx@version has been an integer for the last ten version or so, yes. I will keep it an integer going forward. Did you look at the RELEASE file in the documentation directory? That has a complete explanation of this change plus an example of how to change formats.

PK

On 9 Mar 2016, at 9:57 pm, u-fischer [email protected] wrote:

There are really a lot of people around which uses older system, and you can't write a biblatex style for e.g. a journal and require that all authors use the newest version. Is/was \abx@version always a (decimal) number so that one can safely test it? Then I will implement two versions depending on the value. (But I think that they should be some examples in the docu that show how to convert one definition in the new form).

—
Reply to this email directly or view it on GitHub.

Dr Philip Kime

[u-fischer]

I don't mind jumps, and sometimes incompatible changes are inevitable, but it would be useful to know some time before about them so that one can prepare workarounds.

[plk]

That's a fair point - I did leave the DEV version for testing for many weeks and then waited a couple of weeks after SourceForge upload before TeXLive but I would like to have a mailing list or something for style authors to announce these things - any ideas?

[u-fischer]

Hm, I would naturally read comp.text.tex, but probably a message based on the github account would be better. There could be here an "major change announcement" issue where the interested could comment and so get notifications. (One should probably once write the authors of the style so that they know where to look).

[josephwright]

@plk Could you elaborate on why the underlying change was needed? BibTeX has always had names as 'first part', 'last part', etc.? (I'm trying to work out how I fix my styles, and as @u-fischer says this becomes tricky if there are entirely incompatible formats.)

[josephwright]

Re-reading the docs, I'm a bit confused. The new version has exactly the same data available as the old version that I can see (family, given, prefix, suffix), just accessed in a different way.

[u-fischer]

@josephwright but the structure is obviously now extensible. Also now one no longer has to ponder if #1 or #2 is the first name. Ignoring the problem with the incompatible change I would say it is much cleaner in this way. (But I don't quite understand the \namepart command. I wonder why it isn't executed by default. Also I would probably have named it \getnameparts or \retrievenameparts).

[josephwright]

@u-fischer Sure, but that's a 'with hindsight the code should have been ....' reason, which isn't in itself enough to force a breaking change.

[u-fischer]

Yes, I see problems comming from this change - quite a number of my customers are affected and I found also a number of external biblatex styles which will need to be adapted. Probably it would have been better to introduce a \DeclareExpandedNameFormat.

[hvoss49]

It would have been very nice if such major changes had been announced weeks before in one of the main TeX newsgroups or mailing lists.

[plk]

The point of this is part of the general move to use the datamodel as the source for everything. Too much was hard-coded from old bibtex defaults. The new name stuff allows for arbitrary name parts like patronymics (see the 93-nameparts example file). I know it's pain at the moment but the old bame format code was really a legacy of the ancient idea of names fron 1970s bibtex. This was really hard-coded in the structure of the code and beyond my abilty to make backwards compatible.

[plk]

I did want to announce this and probably should have done - I pondered some sort of mailing list for style authors. The problem is that when I asked for feedback for the biblatex 2.0 release on these forums, I got basically no response and so assumed this wouldn't help much.

[u-fischer]

93-nameparts looks interesting and I think one could (in a better way) solve some problems with japanese and transcriptions I had some time ago. Is there a way to build the name parts with a sourcemap from a bib-file? e.g. from a field "authorpatronymics={XXx and {} and YYY}". (so that one doesn't have to convert everything to xml).

[plk]

That's something I'm thinking about - it is quite tricky because the btparse library parses names in the old bibtex way which is rather inflexible and will never understand anything other than the standard four bibtex name parts. One of the original motivations for biber's tool mode was to make converting to/from biblatexml easy.

[hvoss49]

If every \DeclareNameFormat needs a following \nameparts{#1} the question is why I have to do it manuallly and it is not done by the macro \DeclareNameFormat itself?

[plk]

I am looking at this - there was some technical reason and also user reason that there were some cases when I though this would not be wanted but if this is not really the case, I'll fix it and put in a backwards compatible fix.

[hvoss49]

By the way: I stopped the printing of the second edition of "Bibliografien mit LaTeX" which is already at the printing house. I have 14 days to upload a new pdf. It would be nice that you didn't make other major changes ... ;-)

[plk]

Ok, I'm sorry about this - 3.4 will have some changes too but not as major. I hope the major changes are nearly all done. We really need to think about this going forward, it's the usual OSS problem that when it gets used enough, people (not unreasonably in some respects) expect enterprise-grade release management. With a one-person, completely unpaid development/management team, that's not so easy ...

[josephwright]

@plk I guess what we should do in that regard is the same as the team have in the past: ask people to write good test cases (we can use l3build to run them). Would a blog post be a good plan for that? I can soon write something, perhaps with a demo of how to write.

[hvoss49]

@plk I agree and I know what you are talking about ;-) However, an annoucement, if possible long time before the official release, is a good idea!

[plk]

Yes - and I would really like to create a comprehensive l3build test suite for biblatex too. I have played with it a bit and we’ve corresponded on it but it isn’t currently ideally suited to biblatex testing. I think l3build is a really good thing which we need. The reason why biber is quite stable is because I have a comprehensive test suite setup using one of the many good test modules for Perl and I’ve long wished there were something equivalent for latex.

PK

On 10 Mar 2016, at 2:54 pm, Joseph Wright [email protected] wrote:

@plk I guess what we should do in that regard is the same as the team have in the past: ask people to write good test cases (we can use l3build to run them). Would a blog post be a good plan for that? I can soon write something, perhaps with a demo of how to write.

—
Reply to this email directly or view it on GitHub.

Dr Philip Kime

[plk]

Don't underestimate what "a full set of tests" means though - that's a huge amount of work for all the biblatex functionality ... I have a hacked "full test" at the moment which generates all of the example files for both backends and then runs a perl program to do a character-compare on the resulting PDFs. It is somewhat useful but doesn't tell you anything about which component caused a problem.

[josephwright]

@plk Oh, certainly. The team asked for help with some LaTeX testing in ~1992 and new tests are still needed for things today :-)

[plk]

In biblatex 3.4 (currently on SF), the call to \nameparts{#1} in name formats is no longer required. I can think of some reasons to keep this around for the style-scope ability to inject name parts for non-western names in some use-cases but by default for now, it will be ignored if found.

[josephwright]

@moewew I've gone for biblatex_legacy as this uses no internals and will cope with a re-merger of the code paths

[josephwright]

@hvoss49 That would be a massive issue in terms of stability but also would put a lot of people (including me) off the entire project. BibTeX is much faster than Biber and is rock solid stable: for those of us not needing clever sorting, new features, etc., BibTeX is ideal. (As a user, I want biblatex for the stuff that works fine with both back-ends.)

[hvoss49]

Oh, I suppose you never had a bibliography with czech or japanese or ... names.

[josephwright]

@hvoss49 Like many LaTeX users, I'm in the physical sciences (and mainly using pdfTeX): any accents or whatever are covered by the inputenc mechanisms/LICR and all names are given in Latin characters (it's not been so long that we've even been allowed accents in publications). Moreover, there's no sorting to do in chemistry: we use numbered styles. Some other physical sciences do do sorting but pretty 'easy' versions, which are readily covered using BibTeX.

I don't see my use (as a 'user') as atypical for my broad subject area, though of course I'm well aware that there are lots of things that BibTeX can't do. [To note what I do use from biblatex: the ability to access bibliography data in the document, easy per chapter/section/... bibliographies, etc. Occasionally I do want 'reference sets', which are better with Biber than BibTeX (dynamic with Biber, static-only with BibTeX), but I've increasingly moved away from them as I suspect they are not so helpful to the reader (despite being common in chemistry).]

[plk]

I have some sympathy for both @hvoss49 and @josephwright here. bibtex isn't really being used by biblatex in any real sense, it's being used just for it's incidental sorting capability which is behind the times. It is faster but for a batch program, I'm not sure that matters too much. As far as solodity goes, in the simpler backend use cases which @josephwright mentions, biber is also solid - I have a large test suite of regression tests going back over five years of features which are run before every dev upload. Bugs are fairly rare, especially in core features. It would be nice to have some data on backend use but that's hard to obtain.

[hvoss49]

every 7-Bit program ist fast, so it is not really surprising that bibtex is fast. It is the same for tex->pdftex->xetex->luatex (from fast to slow, or better from fast to not-so-fast ;-)

[plk]

How about a thread on comp.text.tex asking for opinions about dropping bibtex as a backend?

[u-fischer]

@josephwright I'm wouldn't say that you are atypical, but you aren't typical either. In german every bib has to use accented chars in names and other places, and authoryear citations are quite common, so you need reliable utf8-sorting and -handling. Beside this I don't think that speed matter so much in this case. bibtex/biber don't need to run at every compilation. And biber gives better messages when entries are wrong ;-).

[johannesbottcher]

A quite possible example for german. Of course some programs will export pure ascii, but some will have utf8-databases.

\begin{filecontents}{\jobname.bib}
    @book{wombat,
        author = {Bodo Bäcker},
        title = {the wombat stories},
        date = {2016},
    }
    @book{capybara,
        author = {Bernd Bolle},
        title = {capybara},
        date = {2016},
    }
\end{filecontents}
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[
    backend=bibtex,
%   backend=biber,
%   sortlocale=de_DE_phonebook,
style=authoryear]{biblatex}
\addbibresource{\jobname.bib}
\nocite{*}
\begin{document}
\printbibliography
\end{document}

Concerning speed: My machine is so slow that biber sometimes needs several seconds till any terminal output can be seen. Would it be possible to instantly give feedback, for example this is biber? This is the wrong repo though, but it seems to fit right now.

[plk]

I could add a rapid output line but have you tried the --fastsort option?

[johannesbottcher]

Marginal difference

with --fastsort

real    0m3.390s
user    0m3.176s
sys 0m0.148s

without --fastsort

real    0m3.445s
user    0m3.268s
sys 0m0.112s

Maybe it really is time for a faster machine or at least a fresh installation.

[josephwright]

@plk On 'stability' I was not meaning to be critical of Biber. BibTeX is stable in the sense that in the time I've been involved in LaTeX at all there has been one update (v0.99c to v0.99d, for long URLs). There are unlikely to be any further code changes at all, so a file which uses BibTeX today will be processed the same in say 20 years time.

@hvoss49 Sure, and one can argue that by historical standards the issue here is trivial. However, people do mention performance not only of Biber but of other engines vs. pdfTeX, and even of DVI vs. PDF mode, etc.

@u-fischer My BibTeX database does have accents :-) It's notable that a lot of classical BibTeX styles exist and work 'well enough' not only in English but also for other western European languages. I'm not arguing at all that Biber isn't better-able to cope with the complexities here, only that for many uses BibTeX is OK.

The key point for me here is stability. Existing documents using BibTeX as their back-end should continue to process successfully. Biber doesn't always do exactly the same thing as BibTeX for cases that the two can both handle. For example, it will choke on some database entries that BibTeX handles (despite syntax issues). Dropping the back-end entirely would mean breaking that, which arguably should force a new name for the package, etc.: I don't think that is at all desirable (style file sharing would become very complex). Simply 'freezing' the BibTeX code was the original plan to do this, but that then runs into issues when breaking syntax changes are needed from the Biber side.

[johannesbottcher]

I am agreeing with the last paragraph above. If support for BibTeX is frozen, it should be moved into a new package, so that it can be dealt with easily.
Completely dropping BibTeX ... not so good, i think.

[josephwright]

I note @plk has raised the discussion issue on c.t.t.: suggest everyone takes a look (https://groups.google.com/forum/#!topic/comp.text.tex/3tR5B7vDr-w for those people without a Usenet reader).

[u-fischer]

@josephwright I'm not trying to convince you (or someone else) to drop bibtex. I'm only saying that for us german biber is the better choice. Btw: I remember quite well the discussion 3 years ago when it was me who fighted for the bibtex users: e.g. http://newsgroups.derkeiler.com/Archive/Comp/comp.text.tex/2012-04/msg00109.html

[josephwright]

@johannesbottcher It's tricky, though, as sometimes you do have to change things (which is where we came in with this question!). Certainly if I were starting today I'd need a lot of convincing to use BibTeX, in the same way for my LaTeX3 work I'm still very unsure about what to do with pdfTeX (8-bit but again fast and stable) vs. XeTeX/LuaTeX (Unicode, being developed, both not entirely without things to concern the team).

@u-fischer Sure :-) I find myself of course arguing for more work for me: dropping BibTeX entirely would be a much easier path (as a developer), and there is an argument for it, while keeping it working clearly needs some significant effort if the main development is to continue being able to extend the approach.

[johannesbottcher]

@plk Do you have an estimate on the time lapse till something serious changes? I guess this will not happen within the next month, but half a year? One year? I am just asking out of curiosity.

[plk]

Well, given @josephwright's estimate of about two months work to bring the bibtex backend up to date with internal changes etc, this is the time to have the discussion. It all depends on the user feedback really.

[josephwright]

@johannesbottcher As @plk notes, I think I need at least a couple of months to realign the two backend paths: the bulk is easy (unchanged) but getting BibTeX 'right' (enough) needs effort. Certainly if there's a change of status to come I think a long lead would be needed.

[plk]

I'm closing this for now - the issue is known and the thread is a bit general now.