Fixes #4478 - Added API localization #1428

mbacovsky · 2014-05-08T14:03:44Z

This PR contains

localization setup for apipie
rake task for building documentation in parts (index pages and resources documentation)

Doc build duration for reference: Foreman in sec (Foreman + Katello in min:sec) for all supported languages

rake apipie:cache cache_part=resources:      3:00 (5:44)
rake apipie:cache cache_part=index           0:54 (1:48)
rake apipie:cache                            3:24 (6:39)

Resources can be built during package build time and index after package installation.

Known issues:

when installed with katello plugin the index cache rake in posttrans fails silently because katello is not configured (should work after katello-installer is run -> useful for update)
if only indexes are rebuild during rpm install or in foreman-installer all plugins providing API should have the resource docs packaged. If they have not, Foreman API documentation shows '404', but API bindings (Hammer) should not be affected as it use the index only.

Related PR in apipie-rails Apipie/apipie-rails#232
Related PR with API doc string marked for translation #1626

mbacovsky · 2014-05-08T21:14:44Z

The builds are failing on yet unsatisfied dependancy on apipie-rails 0.2.0

mbacovsky · 2014-05-12T10:08:05Z

config/initializers/apipie.rb

@@ -8,6 +8,16 @@
  config.ignored_by_recorder = %w[]
  config.doc_base_url = "/apidoc"
  config.use_cache = Rails.env.production? || File.directory?(config.cache_dir)
+  config.languages = ['en', 'de'] # [] to turn off localized API docs and CLI


I'm not sure how to set the languages properly. Both FastGettext.available_locales and FastGettext.default_available_locales seems to be nil at this point. Any ideas?

Probably the initialiser order, as we set up locales in fast_gettext.rb.

Is there a way to influence the order?

Rename the files as I think they're processed alphabetically.

lzap · 2014-05-12T11:09:59Z

And where do the api texts gets translated eventually?

mbacovsky · 2014-05-12T11:28:41Z

@lzap, that is the bussiness of apipie rake task/controller. The strings are translated during the output to avoid expensive parsing of the sources for each language.

mbacovsky · 2014-05-12T11:56:22Z

@domcleal, renaming fast_gettext.rb to 1_fast_gettext.rb did the trick. The downside is it is taking quite long to generate the cache for all the languages.

@lzap: the typo was (hopefully) fixed

mbacovsky · 2014-05-13T08:27:20Z

[test]

mbacovsky · 2014-05-13T08:27:48Z

Apipie-rails 0.2.0 was released

lzap · 2014-05-13T10:24:35Z

@domcleal, renaming fast_gettext.rb to 1_fast_gettext.rb did the trick. The downside is it is taking quite long to generate the cache for all the languages.

Don't generate for all of them, let's just have a list of "approved"
languages. For now, English and Spanish for example?

Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

mbacovsky · 2014-05-13T11:06:21Z

[test]

mbacovsky · 2014-05-15T20:18:47Z

I've turned off generating docs for locales with some hints in the initializer, and let it up to admin what he sets. Would it make sense to move this to settings and let it set and perhaps trigger the rake apipie:cache from the UI?

I've fixed some tests, but there are still some left, I would use some help with:

the packaging tests are failing on Error: No Package found for rubygem(apipie-rails) >= 0.2.0. So I guess we need to tag the apipie package, to override perhaps?
the test are failing on ruby 2.0, it seems unrelated to my PR, but I'll investigate that further

domcleal · 2014-05-22T14:14:45Z

Wow, you're right, that does take a long time (1 min 35 seconds on fast i7/SSD laptop for rake apipie:cache).

I dislike hard coding a list of languages that are approved, but can see the attractiveness of allowing the admin to choose, a bit like installing "language packs" (maybe we should even package it this way one day). Could we perhaps look at how apipie is actually generating these - is it re-running a large parsing operation for each language rather than parsing once and generating for each language etc?

By the way, I hit an error when mark_translated: true in settings.yaml, as apipie tries to translate nil, these should be filtered out or lib/foreman/gettext/debug.rb needs fixing.

I might also do a quick proof-read of the docs text before merging so it's in good shape for translators.

(As discussed on IRC, we'll sort the RPM failure when coming to merge as apipie-rails 0.2.0's incompatible with develop. Ruby 2.0.0 errors are now fixed.)

mbacovsky · 2014-05-22T14:56:56Z

Yes it is slow, on my laptop it takes approx 80 sec. The first minute is initialization of the rails app and parsing the API controllers. Then it takes about 20 sec per language to generate the JSON and HTML data.
Apipie parses the docs just once and does the translations during the output. When the list of additional languages is empty, the rake task takes almost the same time as before this feature was added to apipie.

I'll look at the failing nils.

Thanks for fixing the issues... 🍺

mbacovsky · 2014-05-27T08:40:21Z

@domcleal I fixed the failures on nil values

mbacovsky · 2014-06-09T13:08:47Z

@domcleal patch was rebased

domcleal · 2014-06-17T12:19:36Z

locale/apipie.rb

@@ -0,0 +1,28 @@
+# Apipie-rails is library we use for ducumenting our API


*documenting

domcleal · 2014-06-17T12:24:12Z

I've proof-read the lot and tried to fix lots of consistency issues - could you squash this in please, if you agree with it? domcleal@6f4843c

The remaining things to sort are:

What to do about enabling/disabling languages: I'm happy to merge this with all languages enabled by default, and then we add a new feature in redmine blocking 1.6 to create some method for enabling/disabling language packs. (Perhaps just packaging.)
Built in apipie-rails text isn't getting translated: mentioned above, inline.
Two controllers (compute_resources, media) include some dynamic stuff inside their strings (lists of CR providers & OS families) which means they won't translate by using # or %. Maybe the config.translate lambda needs special support to replace a magic string with the list of families/CRs, or we just remove the help text?

mbacovsky · 2014-06-19T09:18:25Z

Thanks for fixed strings, I'll squash them in
Re 1. @ohadlevy had an idea to generate the cache during build time, merging the caches from individual plugins should be relatively cheap. I'm experimenting with that right now.
Re 2. answered inline
Re 3. I'll try to fix the lambda. If there is impact on performance, I'll drop the dynamic strings

mbacovsky · 2014-07-03T11:03:02Z

Updated this PR. The changes arein separate commits that I will squash in before merge.
@domcleal 's patch with string and conistency fixes was applied. Lambda was extended with support for dynamic strings. Typos corrected.

All suported languages were enabled in the initializer. Cache now could be pre-generated during build time. With Apipie/apipie-rails#262 we can generate cache for resources separately during build. On installed instance we can regenerate just the index pages with regard on installed plugins. On foreman with katello it takes about 3sec per language plus task initialization which seems acceptable. The downside is that all the plugins needs to be built with pre-generated cache or the plugin cache generation has to be run manually with rake plugin:apipie:cache['plugin-name'] cache_part=resources .

$ time bundle exec rake apipie:cache cache_part=index
Workaround for RbVmomi may not work as ComputeResource is already loaded: ComputeResource
[deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
2014-07-03 10:42:17 +0000 | Started
2014-07-03 10:42:39 +0000 | Documents loaded...
2014-07-03 10:42:39 +0000 | Processing docs for 
2014-07-03 10:42:43 +0000 | Processing docs for ja
2014-07-03 10:42:46 +0000 | Processing docs for zh_CN
2014-07-03 10:42:49 +0000 | Processing docs for en_GB
2014-07-03 10:42:52 +0000 | Processing docs for es
2014-07-03 10:42:55 +0000 | Processing docs for gl
2014-07-03 10:42:58 +0000 | Processing docs for de
2014-07-03 10:43:01 +0000 | Processing docs for en
2014-07-03 10:43:04 +0000 | Processing docs for sv_SE
2014-07-03 10:43:07 +0000 | Processing docs for fr
2014-07-03 10:43:10 +0000 | Finished

real    1m10.356s
user    0m0.000s
sys     0m0.000s

For the rake task to know the plugin's API controllers I extended foreman plugin with two attrs.

If this will be found viable I can fix the spec files and installer.

The only not addressed thing still remains apipie internal strings, which will follow

mbacovsky · 2014-07-03T11:04:00Z

The PR reflecting plugin changes in katello is here Katello/katello#4375

mbacovsky · 2014-07-03T11:38:51Z

rebased

lzap · 2014-07-04T18:26:33Z

All suported languages were enabled in the initializer. Cache now
could be pre-generated during build time. With
Apipie/apipie-rails#262 we can generate cache
for resources separately during build. On installed instance we can
regenerate just the index pages with regard on installed plugins. On
foreman with katello it takes about 3sec per language plus task
initialization which seems acceptable. The downside is that all the
plugins needs to be built with pre-generated cache or the plugin cache
generation has to be run manually with rake plugin:apipie:cache['plugin-name'] cache_part=resources .

Hmmm, lgtm. Can you raise a PR to the foreman-packaging repo for the
foreman.spec change, and to any plugin spec as an example together with
scratch builds?

Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

dLobatog · 2014-07-22T10:08:55Z

@mbacovsky rebase needed

domcleal · 2014-07-29T08:59:48Z

@mbacovsky it may help to split this PR into two - one that simply adds API strings for extraction so we can get them into Transifex and avoid more rebasing & conflicts with other PRs, and then keep this open for the apipie and plugins work. I'd be happy to merge the extractions ASAP as I think it'll make our lives easier and it has no other impact.

mbacovsky · 2014-07-29T14:43:35Z

@domcleal I moved the string translations to separate PR #1626 as I'm still fighting with the packaging work for this PR

mbacovsky · 2014-11-04T13:06:28Z

@lzap, @domcleal, updated, rebased. PR description updated, known issues added there. API cache is now build for all supported languages. Part is build during build time and part after installation. Rough times for comparsion are in the PR description. Katello PR was updated. foreman-packaging PR was added theforeman/foreman-packaging#426

lzap · 2014-11-05T08:55:39Z

Rubocop failure.

mbacovsky · 2014-11-05T09:30:44Z

Rubocop issues fixed

dLobatog · 2014-11-12T10:08:11Z

Merged as a59972c thanks @mbacovsky!
theforeman/foreman-packaging#426 pending

domcleal added Needs testing labels May 8, 2014

mbacovsky mentioned this pull request May 8, 2014

Refs #4478 - Add support for localized API docs Katello/katello#4063

Merged

mbacovsky reviewed May 12, 2014
View reviewed changes

domcleal reviewed Jun 17, 2014
View reviewed changes

locale/apipie.rb

@@ -0,0 +1,28 @@

# Apipie-rails is library we use for ducumenting our API

Copy link

Contributor

domcleal Jun 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*documenting

domcleal added Waiting on contributor and removed Not yet reviewed labels Jun 17, 2014

mbacovsky mentioned this pull request Jul 21, 2014

DO NOT MERGE fixes #4181 - change API V2 functional POST/PUT tests and apipie definitions to reflect unwrapped params (no root node) #1601

Closed

iNecas mentioned this pull request Jul 23, 2014

DO NOT MERGE fixes #6749 - regenerate API docs, fix API v2 examples in docs that show error messages theforeman/theforeman.org#237

Closed

mbacovsky force-pushed the 4478_localized_api branch from d16dcc2 to a39f1e1 Compare November 3, 2014 14:50

mbacovsky mentioned this pull request Nov 4, 2014

Refs #4478 - prebuild localized apipie cache theforeman/foreman-packaging#426

Closed

lzap added Ready to merge and removed Waiting on contributor labels Nov 5, 2014

Fixes theforeman#4478 - API documentation localized

9920782

mbacovsky force-pushed the 4478_localized_api branch from a39f1e1 to 9920782 Compare November 5, 2014 09:07

dLobatog closed this Nov 12, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #4478 - Added API localization #1428

Fixes #4478 - Added API localization #1428

mbacovsky commented May 8, 2014

mbacovsky commented May 8, 2014

mbacovsky May 12, 2014

domcleal May 12, 2014

mbacovsky May 12, 2014

domcleal May 12, 2014

lzap commented May 12, 2014

mbacovsky commented May 12, 2014

mbacovsky commented May 12, 2014

mbacovsky commented May 13, 2014

mbacovsky commented May 13, 2014

lzap commented May 13, 2014

mbacovsky commented May 13, 2014

mbacovsky commented May 15, 2014

domcleal commented May 22, 2014

mbacovsky commented May 22, 2014

mbacovsky commented May 27, 2014

mbacovsky commented Jun 9, 2014

domcleal Jun 17, 2014

domcleal commented Jun 17, 2014

mbacovsky commented Jun 19, 2014

mbacovsky commented Jul 3, 2014

mbacovsky commented Jul 3, 2014

mbacovsky commented Jul 3, 2014

lzap commented Jul 4, 2014

dLobatog commented Jul 22, 2014

domcleal commented Jul 29, 2014

mbacovsky commented Jul 29, 2014

mbacovsky commented Nov 4, 2014

lzap commented Nov 5, 2014

mbacovsky commented Nov 5, 2014

dLobatog commented Nov 12, 2014

		@@ -0,0 +1,28 @@
		# Apipie-rails is library we use for ducumenting our API

Fixes #4478 - Added API localization #1428

Fixes #4478 - Added API localization #1428

Conversation

mbacovsky commented May 8, 2014

mbacovsky commented May 8, 2014

mbacovsky May 12, 2014

Choose a reason for hiding this comment

domcleal May 12, 2014

Choose a reason for hiding this comment

mbacovsky May 12, 2014

Choose a reason for hiding this comment

domcleal May 12, 2014

Choose a reason for hiding this comment

lzap commented May 12, 2014

mbacovsky commented May 12, 2014

mbacovsky commented May 12, 2014

mbacovsky commented May 13, 2014

mbacovsky commented May 13, 2014

lzap commented May 13, 2014

mbacovsky commented May 13, 2014

mbacovsky commented May 15, 2014

domcleal commented May 22, 2014

mbacovsky commented May 22, 2014

mbacovsky commented May 27, 2014

mbacovsky commented Jun 9, 2014

domcleal Jun 17, 2014

Choose a reason for hiding this comment

domcleal commented Jun 17, 2014

mbacovsky commented Jun 19, 2014

mbacovsky commented Jul 3, 2014

mbacovsky commented Jul 3, 2014

mbacovsky commented Jul 3, 2014

lzap commented Jul 4, 2014

dLobatog commented Jul 22, 2014

domcleal commented Jul 29, 2014

mbacovsky commented Jul 29, 2014

mbacovsky commented Nov 4, 2014

lzap commented Nov 5, 2014

mbacovsky commented Nov 5, 2014

dLobatog commented Nov 12, 2014