[AIRFLOW-5065] Add colors to console log #5681
Conversation
|
How does it look in Web UI? Very useful feature. This has long been missing for me. Now logs are very unreadable. |
|
In UI you can spot some ASCII codes 😕 |
There was a problem hiding this comment.
I like it a lot - having coloured error logs in console might be really helpful in detecting errors. I just wonder if it can have some @ashb @BasPH @feluelle @kaxil -> what do you think - is there any reason console log should not be coloured for compatibility ? I think any automation of the logs uses Files not reading from stdout so we should be rather safe ?
We can check if stdout has terminal and only use colours if it has @nuclearpinguin : |
@potiuk I imagine the logs to be read by humans and thus coloured logs are a big plus. |
I think the logic behind adding colors ( |
|
I'll look at this tomorrow. Like the idea, but I wonder if we can do something like git does (where you have |
|
I made a quick check - and I think indeed we could use for example this: https://github.com/borntyping/python-colorlog instead of our custom Formatter. It will then be more configurable for the user. Possibly we should consider using it instead. I wonder if this has any disadvantages @nuclearpinguin ? |
|
@potiuk Does this library write arguments using a different text style? I think not. One request from my side was that the arguments should be written using a different text style. |
|
Some of my thoughts:
|
|
@mik-laj @nuclearpinguin -> I have no idea if it supports it, that's a good question. But it would be worthwhile to check if this (or other library) already provides similar or the same features. I think it's a good question just raised - whether customisable colours in logging (configured via string as Ash mentioned) is more important that our own "opinionated" way of colouring. And maybe we do not have to choose - if one of the ready-to-use libraries already provides enough flexibility. Just a search away and I also found another library for that, which seems super-customisable. https://coloredlogs.readthedocs.io/en/latest/api.html#examples-of-customization. I am not saying one approach is better than the other. There is a value in both approaches, but it would be great to know that we considered alternatives (especially that they are easily available), and why we chose this one. |
|
from colorlog import ColoredFormatter
import logging
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
log_format = '[%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s%(reset)s:%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(message_log_color)s%(message)s'
formatter = ColoredFormatter(
log_format,
log_colors={
'WARNING': 'yellow',
'ERROR': 'red',
'CRITICAL': 'red,bold',
},
secondary_log_colors={
'message': {
'INFO': 'bold',
}
},
)
logging.root.handlers[0].setFormatter(formatter)
log.info("Usiung connection to: %s", {'id': 'proxy_postgres_tcp'})
log.warning("Hi")
log.error("Hi")
try:
raise RuntimeError("x")
except Exception:
log.critical("err")Looks like:
And it automatically detects when in a TTY. But it may not be written in a way that lets us get the colours we want. |
|
Both coloredlogs and colorlog has no out of the box option for formatting message arguments and exception traceback(?), that's why I decided to make a custom formatter. Additionally my implementation works only on LogRecord thus not modifying message format as in case of To use Personally I really like the idea of changing color / boldness / background of attributes in log messages. But this could be easily added aby extending |
|
Not changing the LogRecord does seem like a good idea. |
turbaszek
force-pushed
the
colored-log
branch
2 times, most recently
from
1222cc7 to
68ccde1
Compare
|
Finally I decided to extend '[%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s%(reset)s:%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(message_log_color)s%(message)s'Now two dependencies are required: |
Checked and those are both MIT so that's cool. |
|
Confirmed task logs look okay now, but I noticed that we have one file being written with ascii escapes in: I think we can hack around that by adding |
|
I pushed a fixup commit with this change directly to your branch, ack but that now conflicts. I'll fix that too. |
This commit adds custom formatter for Airflow console log.
Thanks! |
|
Boom, all green, merging! |
This commit adds custom formatter for Airflow console log to display colours when connected to a TTY (cherry picked from commit 2d7b1a0)
This commit adds custom formatter for Airflow console log to display colours when connected to a TTY (cherry picked from commit 2d7b1a0)
This commit adds custom formatter for Airflow console log to display colours when connected to a TTY (cherry picked from commit 2d7b1a0)
|
Since 1.10.4, some ANSCI escape characters are showing up in the Web UI like this: Any idea how to suppress these? |
This commit adds custom formatter for Airflow console log.
Make sure you have checked all steps below.
Jira
Description
Tests
Commits
Documentation
Code Quality
flake8