Tutorials#

Here are a few tutorials with different difficulty levels.

Beginners will want to start with the blog building tutorial giving a short introduction to the basic concepts. Then the photo gallery construction tutorial highlights more advanced concepts such as unit tests, security settings and migration scripts.

The other tutorials cover specific topics you can learn about when you understand the basics.

Building a simple blog with CubicWeb#

CubicWeb is a semantic web application framework which favors reuse and object-oriented designs.

This tutorial is designed to help you make your very first steps with CubicWeb. It will guide you through basic concepts such as:

  • getting an application running by using existing components

  • discovering the default user interface

  • extending and customizing the look and feel of that application

More advanced concepts are covered in Building a photo gallery with CubicWeb.

Some vocabulary#

CubicWeb comes with a few words of vocabulary that you should know to understand what we’re talking about. To follow this tutorial, you should at least know that:

  • a cube is a component that usually includes a model defining some data types and a set of views to display them. A cube can be built by assembling other cubes;

  • an instance is a specific installation of one or more cubes and includes configuration files, a web server and a database.

Reading The Core Concepts of CubicWeb for more vocabulary will be required at some point.

Now, let’s start the hot stuff!

Get a blog running in five minutes!#

First choose and follow the installation method of your choice.

Once you have CubicWeb setup, install the blog cube using the following command:

pip install cubicweb-blog

Then you can create and initialize your blog instance:

cubicweb-ctl create blog myblog

Here the blog argument tells the command to use the blog cube as a base for your instance named myblog.

Note

If you get a permission error of the kind OSError: [Errno 13] Permission denied: '/etc/cubicweb.d/myblog' , read the next section.

This command will ask you a series of question. The first one is about the database engine to use (SQLite or PostgreSQL). For this tutorial, we will use SQLite as it is easier to setup and does not need a database server. In production environments, PostgreSQL is recommended as it offers better performances. More information on database configuration can be found here.

The command will also create a user used to manage your instance, for which you will be asked to give a name and password.

You can leave the remaining questions to their default by simply pressing Enter.

Note

If you get errors during installation such as:

while handling language es: [Errno 2] No such file or directory: 'msgcat': 'msgcat'
while handling language en: [Errno 2] No such file or directory: 'msgcat': 'msgcat'
while handling language fr: [Errno 2] No such file or directory: 'msgcat': 'msgcat'

This means you are missing the gettext dependency. To fix this, follow the instructions in the section Installing Dependencies. Then either restart the installation process or run cubicweb-ctl i18ncubicweb && cubicweb-ctl i18ncube blog after installation. More information in Internationalization.

Then you need to tell CubicWeb your instance is going to run on the localhost by editing ~/etc/cubicweb.d/myblog/all-in-one.conf. In this file under the [MAIN] section, replace the line #host=` by `host=localhost.

Once this process is complete (including database initialisation), you can start your instance by using:

cubicweb-ctl pyramid -D myblog

The -D option activates the debugging mode. Removing it will launch the instance as a daemon in the background.

This is it, your blog is functional and running at http://localhost:8080!

About file system permissions#

Unless you installed from source, the above commands will initialize your instance as a regular user in your home directory (under ~/etc/cubicweb.d/). If you installed from source, your instance will be created in system directories and thus will require root privileges. To change this behavior, please have a look at the Resource mode section.

Instance parameters#

If you would like to change database parameters such as the database host or the user name used to connect to the database, edit the sources file located in the /etc/cubicweb.d/myblog directory.

Then relaunch the database creation:

cubicweb-ctl db-create myblog

Other parameters, like web server or emails parameters, can be modified in the /etc/cubicweb.d/myblog/all-in-one.conf file (or ~/etc/cubicweb.d/myblog/all-in-one.conf depending on your configuration.)

You’ll have to restart the instance after modification in one of those files.

Discovering the web interface#

You can now access your web instance to create blogs and post messages by visiting the URL http://localhost:8080.

By default, anonymous access is disabled, so a login form will appear.

the login form

If you asked to allow anonymous access when initializing the instance, click on the ‘login’ link in the top right hand corner. To login, you need to use the admin account you specified at the time you initialized the database with cubicweb-ctl create.

the index page with anonymous access

Once authenticated, you can start playing with your instance. You will notice the index page has changed compared to the anonymous access view. There are more entries in the Manage section and and some new [+] buttons have appeared next to the entities. These allow you to edit and add new entries in the database.

the index page when logged in

Note

If you find untranslated strings such as blog.latest_blogs in the sidebar:

the index page with anonymous access

This means you are missing the gettext dependency. To fix this, follow the instructions in the section Installing Dependencies. Then either restart the installation process or run cubicweb-ctl i18ncubicweb && cubicweb-ctl i18ncube blog after installation. More information in Internationalization.

Minimal configuration#

Before creating entities, let’s change the unset title string in the header. This string is set by a CubicWeb system properties and represents the site’s title. To modify it, click on the site configuration link in the Manage section.

This will open a new page with different categories. You will find the site’s title in the ui section. Simply set it to the desired value and click the ‘button_ok’ button.

the site configuration form

You should see a changes applied message in green at the top of the section. You can now go back to the index page by clicking on the CubicWeb logo in the upper left-hand corner.

You will much likely still see unset title at this point. This is because by default the index page is cached for performance reasons. Force a refresh of the page (Ctrl-R in Firefox) and you should now see the title you entered.

Adding entities#

The blog cube defines several entity types. For example, the Blog entity is a container for a BlogEntry (i.e. posts) on a particular topic. We can get a graphical view of the schema by clicking on the data model schema link in the Manage section of the index page:

graphical view of the schema (aka data-model)

Note

If you get the error FileNotFoundError: [Errno 2] File not found: dot: 'dot' when accessing the page, this means you are missing the package graphviz. To fix this, follow the instructions in the section Installing Dependencies.

missing graphviz error

Notice that like most other things we will see in this tutorial, this schema is generated by the framework according to the application’s model. In our case the model is defined by the blog cube.

Now let’s create a few of those entities.

Adding a blog#

Clicking on the [+] at the left of the Blog link on the index page will open an HTML form to create a new blog.

the blog creation form

For instance, call this new blog Tech-blog and type in everything about technology as the description , then validate the form by clicking on validate. You will be redirected to the primary view of the newly created blog.

the blog primary view
Adding a blog post#

There are several ways to add a blog entry. The simplest is to click on the add blog entry link in the actions box on the left while viewing the blog you just created. You will then see a form to create a post, with a blog entry of field preset to the blog you are coming from. Enter a title, some content, click the validate button and you’re done. You will be redirected to the blog’s primary view, though you now see that it contains the blog post you have just created.

the blog primary view after creation of a post

Notice how some new items appeared in the left column.

You can achieve the same result by clicking on the [+] at the left of the Blog entry link on the index page. Since there is no context information, the blog entry of selector will not be preset to a blog if you have more than one.

If you click on the modify link in the action box, you will be taken back to the form to edit the entity you just created. But the form will now have another section with a combo-box entitled add relation providing a generic way to edit relations. Choose the relation you want to add and a second combo box will appear where you can pick existing entities. If there are too many of them, you will be offered to navigate to the target entity. This will open a new page and you will be taken back to your form once you have selected an entity.

the generic relations combo box

This combo-box cannot appear until the entity is actually created, explaining why you could not see it at creation time using the first form. Another way to show this combo-box is to hit apply instead of validate to create the entity without closing the form.

About UI auto-adaptation#

One of the things making CubicWeb different from other frameworks is its automatic user interface adapting itself according to the data being displayed. Let’s see an example.

If you go back to the home page and click on the Blog link, you will be redirected to the blog’s primary view as we have seen earlier. Now add another blog, go back to the index page, and click again on this link. You will see a very different view (namely the list view).

the list view when there are more than one blog to display

In the first case the framework chose to use the primary view since there was only one entity in the data to be displayed. Now that there are two entities, the list view is more appropriate and hence is being used.

There are various other places where CubicWeb adapts to display data in the best way, the main being provided by the view selection mechanism that will be detailed later.

Digging deeper#

By following the principles explained above you should now be able to create new users for your application and to configure your instance. You will notice that the index page lists a lot of types we did not talk know about. Most are built-in types provided by the framework to make the whole system work. You may ignore them in a first time and discover them as time goes.

One thing that is worth playing with is the search box. It may be used in various ways, from simple full text search to advanced queries using the RQL syntax .

Customizing your application#

Usually you won’t get enough by assembling cubes out-of-the-box. You will want to customize them to get personal look and feel, add your own data model and so on. Or maybe start from scratch?

So let’s get a bit deeper and start coding our own cube. In our case, we want to customize the blog we created to add more features to it.

Creating your own cube#

Once your CubicWeb development environment is set up, you can create a new cube:

cubicweb-ctl newcube mycube

This will create a a directory named cubicweb-mycube reflecting the structure described in Standard structure for a cube.

To install your new cube on the virtual environment created previously, run the following command in cubicweb-mycube directory:

pip install -e .

All cubicweb-ctl commands are described in details in cubicweb-ctl tool.

Cube metadata#

The folder cubicweb_mycube/ contains the actual code and metadata for your cube. In this folder, a simple set of metadata about your cube are stored in the __pkginfo__.py file. In our case, we want to extend the blog cube, so we have to tell that our cube depends on this cube by modifying the __depends__ dictionary in that file:

__depends__ =  {"cubicweb": ">= 3.35.0", "cubicweb-blog": None}

where None means we do not depend on a particular version of the cube.

Extending the data model#

The data model or schema is the core of your CubicWeb application. It defines the type of content your application will handle. It is defined in the file schema.py of the cube.

Defining our model#

Let’s say we want a new entity type named Community with a name and a description. A Community will hold several blogs.

We can edit the schema.py as follows:

from yams.buildobjs import EntityType, RelationDefinition, String, RichString

class Community(EntityType):
    name = String(maxsize=50, required=True)
    description = RichString()

class community_blog(RelationDefinition):
    subject = 'Community'
    object = 'Blog'
    cardinality = '*?'
    composite = 'subject'

The import from the yams package provides necessary classes to build the schema.

This file defines the following:

  • a Community has a name and a description as attributes

    • the name is a string which is required and cannot be longer than 50 characters

    • the description is an unconstrained string and may contains rich content such as HTML or Restructured text.

  • a Community may be linked to a Blog using the community_blog relation

    • * means a community may be linked from 0 to N blog, ? means a blog may be linked to 0 to 1 community. For completeness, you can also use + for 1 to N, and 1 for a single mandatory relation (e.g. one to one);

    • this is a composite relation where Community (e.g. the subject of the relation) is the composite. That means that if you delete a community, its blog will be deleted as well.

Of course, there are a lot of other data types and relations such as constraints, permissions, etc, that may be defined in the schema but those will not be covered in this tutorial.

Notice that our schema refers to the Blog entity type which is not defined here. But we know this type is available since we depend on the blog cube defining it.

Applying changes from the model into our instance#

The problem is that we created an instance using the blog cube, not our mycube cube. If we do not do anything there is no way we’ll see anything changing in the myblog instance.

As we do not have any really valuable data in the instance, an easy way would be to trash it and recreated it. First stop the running instance by pressing Ctrl-C in the terminal running the server in debug mode. Then run the following commands:

cubicweb-ctl delete myblog
cubicweb-ctl create mycube myblog
cubicweb-ctl pyramid -D myblog

Another way is to add our cube to the instance using the cubicweb-ctl shell facility. It is a python shell connected to the instance with some special commands available to manipulate it (the same as you’ll have in migration scripts, which are not covered in this tutorial). In that case, we are interested in the add_cube command. First stop the instance by pressing Ctrl-C in the terminal running the server in debug mode and enter the shell using the following command:

cubicweb-ctl shell myblog

Then in the python shell, type the add_cube command:

add_cube('mycube')

Press Ctrl-D to exit then restart your instance:

cubicweb-ctl pyramid -D myblog

The add_cube command is enough since it automatically updates our application to the cube’s schema. There are plenty of other migration commands of a more finer grain. They are described in Migration

If you take another look at the schema on your instance, you will see that changes to the data model have actually been applied (meaning database schema updates and all necessary actions have been done).

the instance schema after adding our cube

If you follow the Site information link in the home page, you will also see that the instance is using blog and mycube cubes (sioc is a dependency of the blog cube).

the instance schema after adding our cube

You can now add some communities and link them to a blog. You will see that the framework provides default views for this entity type (we have not yet defined any view for it!), and also that the blog primary view will show the community it is linked to if any. All this thanks to the model driven interface provided by the framework.

We will now see how to redefine each of them according to your needs and preferences.

Defining your views#

CubicWeb provides a lot of standard views in the directory cubicweb/web/views/. We already talked about primary and list views, which are views applying to one or more entities.

A view is defined by a python class which includes:

  • an identifier: all objects used to build the user interface in CubicWeb are recorded in a registry and this identifier will be used as a key in that registry to store the view. There may be multiple views for the same identifier.

  • a selector, which is a kind of filter telling how well a view suits to a particular context. When looking for a particular view (e.g. given an identifier), CubicWeb computes for each available view with that identifier a score which is returned by the selector. Then the view with the highest score is used. The standard library of predicates is in cubicweb.predicates.

A view has a set of methods inherited from the cubicweb.view.View class, though you do not usually derive directly from this class but from one of its more specific child class.

Last but not least, CubicWeb provides a set of default views accepting any kind of entities.

To illustrate this, we will create a community as we already have done for other entity types through the index page. You will get a screen similar to this:

the default primary view for our community entity type
Changing the layout of the application#

The layout is the general organization of the pages in the website. Views generating the layout are sometimes referred to as templates. They are implemented by the framework in the module cubicweb.web.views.basetemplates. By overriding classes in this module, you can customize whatever part you wish of the default layout.

CubicWeb provides many other ways to customize the interface thanks to actions and components (which you can individually (de)activate, control their location, customize their look…) as well as “simple” CSS customization. You should first try to achieve your goal using such fine grained parametrization rather then overriding a whole template, which usually embeds customisation access points that you may loose in the process.

But for the sake of example, let’s say we want to change the generic page footer. We can simply add in the file cubicweb_mycube/views.py the code below:

from cubicweb.web.views import basetemplates


class MyHTMLPageFooter(basetemplates.HTMLPageFooter):

    def footer_content(self):
        self.w(u'This website has been created with <a href="http://cubicweb.org">CubicWeb</a>.')


def registration_callback(vreg):
    vreg.register_all(globals().values(), __name__, (MyHTMLPageFooter,))
    vreg.register_and_replace(MyHTMLPageFooter, basetemplates.HTMLPageFooter)

  • Our class inherits from the default page footer to ease getting things right, but this is not mandatory.

  • When we want to write something to the output stream, we simply call self.w, which must be passed a unicode string.

  • Since both HTMLPageFooter and MyHTMLPageFooter have the same selector, hence the same score the framework would not be able to choose which footer to use. In this case we want our footer to replace the default one, so we have to define a registration_callback() function to control object registration. The first instruction tells to register everything in the module but the MyHTMLPageFooter class, then the second to register it instead of HTMLPageFooter. Without this function, everything in the module is registered blindly.

Note

When a view is modified while running in debug mode, it is not required to restart the instance server. Save the Python file and reload the page in your web browser to view the changes.

You will now see this simple footer on every page of the website.

Primary view customization#

The primary view (i.e. any view with the identifier set to primary) is the one used to display all the information about a single entity. The standard primary view is one of the most sophisticated views of all. It has several customisation points, but its power comes with uicfg allowing you to control it without having to subclass it.

However this is a bit off-topic for this first tutorial. Let’s say we simply want a custom primary view for the Community entity type, using directly the view interface without trying to benefit from the default implementation (you should do that though if you’re rewriting reusable cubes; everything is described in more details in The Primary View).

here is the code that we will put in the file cubicweb_mycube/views.py of our cube:

from cubicweb.predicates import is_instance
from cubicweb.web.views import primary


class CommunityPrimaryView(primary.PrimaryView):
    __select__ = is_instance('Community')

    def cell_call(self, row, col):
        entity = self.cw_rset.get_entity(row, col)
        self.w(u'<h1>Welcome to the "%s" community</h1>' % entity.printable_value('name'))

        if entity.description:
            self.w(u'<p>%s</p>' % entity.printable_value('description'))

What’s going on here?

  • Our class inherits from the default primary view, here mainly to get the correct view identifier, since we do not use any of its features.

  • We set on it a selector telling that it only applies when trying to display some entity of the Community type. This is enough to get an higher score than the default view for entities of this type.

  • A view that applies to an entity usually has to define the method cell_call as an entry point. This receives the arguments row and col telling to which entity in the result set the view is applied. We can then get this entity from the result set (self.cw_rset) by using the get_entity method.

  • To ease thing, we access our entity’s attribute to display using its printable_value method, which will handle formatting and escaping when necessary. As you can see, you can also access attributes by their name on the entity to get the raw value.

You can now reload the page of the community we just created and see the changes.

the custom primary view for our community entity type

We have seen here a lot of thing you will have to deal with to write views in CubicWeb. The good news is that this is almost everything that is used to build higher level layers.

Note

As things get complicated and the volume of code in your cube increases, you can of course still split your views module into a python package with subpackages.

You can find more details about views and selectors in Principles.

Write entities to add logic in your data#

CubicWeb provides an ORM (Object-Relational Mapper) to programmatically manipulate entities (just like the one we have fetched earlier by calling get_entity on a result set). By default, entity types are instances of the AnyEntity class, which holds a set of predefined methods as well as properties automatically generated for attributes/relations of the type it represents.

You can redefine each entity to provide additional methods or whatever you want to help you write your application. Customizing an entity requires that your entity:

  • inherits from cubicweb.entities.AnyEntity or any subclass

  • defines a __regid__ linked to the corresponding data type of your schema

You may then want to add your own methods, override default implementation of some method, etc… To do so, write this code in mycube/entities.py:

from cubicweb.entities import AnyEntity, fetch_config


class Community(AnyEntity):
    """customized class for Community entities"""
    __regid__ = 'Community'

    fetch_attrs, cw_fetch_order = fetch_config(['name'])

    def dc_title(self):
        return self.name

    def display_cw_logo(self):
        return 'CubicWeb' in self.name

In this example:

  • we used the fetch_config() convenience function to tell which attributes should be prefetched by the ORM when looking for some related entities of this type, and how they should be ordered

  • we overrode the standard dc_title() method, used in various place in the interface to display the entity (though in this case the default implementation would have had the same result)

  • we implemented here a method display_cw_logo() which tests if the community title contains CubicWeb. It can then be used when you are writing code involving Community entities in your views, hooks, etc. For instance, you can modify your previous views as follows:

class CommunityPrimaryView(primary.PrimaryView):
    __select__ = is_instance('Community')

    def cell_call(self, row, col):
        entity = self.cw_rset.get_entity(row, col)
        self.w(u'<h1>Welcome to the "%s" community</h1>' % entity.printable_value('name'))

        if entity.display_cw_logo():
            self.w(u'<img src="https://docs.cubicweb.org/_static/logo-cubicweb.svg"/>')

        if entity.description:
            self.w(u'<p>%s</p>' % entity.printable_value('description'))

Then each community whose description contains ‘CW’ is shown with the CubicWeb logo in front of it.

Note

As for view, you don’t have to restart your instance when modifying some entity classes while your server is running in debug mode, the code will be automatically reloaded.

Extending the application by using more cubes!#

One of the goals of the CubicWeb framework is to have truly reusable components. To do so they must behave nicely when plugged into the application and be easily customisable, from the data model to the user interface. Thanks to systems such as the selection mechanism and the choice to write views as python code, we can build our pages using true object oriented programming techniques to achieve this goal.

A library of standard cubes is available at the CubicWeb Forge to address a lot of common problems such as manipulating files, people, todos, etc. In our community blog case, we could be interested for instance in functionalities provided by the comment and tag cubes. comment provides threaded discussion functionalities and tag a simple tag mechanism to classify content. We will first modify our cube’s __pkginfo__.py file to add those cubes as dependencies:

__depends__ =  {'cubicweb': '>= 3.35.0',
                'cubicweb-blog': None,
                'cubicweb-comment': None,
                'cubicweb-tag': None}

Now we will simply tell on which entity types we want to activate the comment and tag cubes by adding respectively the comments and tags relations on them in our schema (schema.py).

class comments(RelationDefinition):
    subject = 'Comment'
    object = 'BlogEntry'
    cardinality = '1*'
    composite = 'object'


class tags(RelationDefinition):
    subject = 'Tag'
    object = ('Community', 'BlogEntry')

In the above code we activated comments on BlogEntry entities and tags on both Community and BlogEntry. Various views from both comment and tag cubes will then be automatically displayed when one of those relations is supported.

Let’s install the cubes and synchronize the data model as we’ve done earlier. So first install the cubes:

pip install cubicweb-comment cubicweb-tag

Stop the instance by pressing Ctrl-C in the terminal running the server in debug mode and enter the migration shell:

cubicweb-ctl shell myblog

Add the new cubes and exit with Ctrl-D:

add_cubes(('comment', 'tag'))

Then restart the instance with cubicweb-ctl pyramid -D myblog and open a blog entry:

the primary view for a blog entry with comments and tags activated

As you can see, we now have a box displaying tags and a section proposing to add a comment and displaying existing one below the post. All this without changing anything in our views, thanks to the design of generic views provided by the framework. Though if we take a look at a community, we will not see the tags box! This is because by default this box tries to locate itself in the right column within the white frame, and this column is handled by the primary view we overrode. Let’s change our view to make it more extensible, by keeping both our custom rendering but also extension points provided by the default implementation.

Add the following code in cubicweb_mycube/views.py:

class CommunityPrimaryView(primary.PrimaryView):
    __select__ = is_instance('Community')

    def render_entity_title(self, entity):
        self.w(u'<h1>Welcome to the "%s" community</h1>' % entity.printable_value('name'))

    def render_entity_attributes(self, entity):
        if entity.display_cw_logo():
            self.w(u'<img src="https://docs.cubicweb.org/_static/logo-cubicweb.svg"/>')

        if entity.description:
            self.w(u'<p>%s</p>' % entity.printable_value('description'))

By reloading the Community page, it will now appear properly:

the custom primary view for a community entry with tags activated

You can control part of the interface independently from each others, piece by piece.

What’s next?#

In this tutorial we have seen you can build a web application in a few minutes simply by defining a data model. You get a working application which you can then customize without breaking your workflow. You can show results to customers right from the beginning to make the right decisions early in the process. This is important in agile development practices.

The next steps will be to discover hooks, security, data sources, digging deeper into view writing and interface customisation… Still a lot of fun stuff to discover! You will find more tutorials and howtos in the blog published on the CubicWeb.org website.

Use Windmill with CubicWeb#

Windmill implements cross browser testing, in-browser recording and playback, and functionality for fast accurate debugging and test environment integration.

Online features list is available.

Installation#

Windmill#

You have to install Windmill manually for now. If you’re using Debian, there is no binary package (yet).

The simplest solution is to use a setuptools/pip command (for a clean environment, take a look to the virtualenv project as well):

$ pip install windmill
$ curl -O http://github.com/windmill/windmill/tarball/master

However, the Windmill project doesn’t release frequently. Our recommandation is to used the last snapshot of the Git repository:

$ git clone git://github.com/windmill/windmill.git HEAD
$ cd windmill
$ python3 setup.py develop

Install instructions are available.

Be sure to have the windmill module in your PYTHONPATH afterwards:

$ python3 -c "import windmill"

X dummy#

In order to reduce unecessary system load from your test machines, It’s recommended to use X dummy server for testing the Unix web clients, you need a dummy video X driver (as xserver-xorg-video-dummy package in Debian) coupled with a light X server as Xvfb.

The dummy driver is a special driver available with the XFree86 DDX. To use the dummy driver, simply substitue it for your normal card driver in the Device section of your xorg.conf configuration file. For example, if you normally uses an ati driver, then you will have a Device section with Driver “ati” to let the X server know that you want it to load and use the ati driver; however, for these conformance tests, you would change that line to Driver “dummy” and remove any other ati specific options from the Device section.

From: http://www.x.org/wiki/XorgTesting

Then, you can run the X server with the following command

$ /usr/bin/X11/Xvfb :1 -ac -screen 0 1280x1024x8 -fbdir /tmp

Windmill usage#

Record your use case#

  • start your instance manually

  • start Windmill with url site as last argument (read Usage or use ‘-h’ option to find required command line arguments)

  • use the record button

  • click on save to obtain python code of your use case

  • copy the content to a new file in a windmill directory

If you are using firefox as client, consider the “firebug” option.

If you have a running instance, you can refine the test by the loadtest windmill option:

$ windmill -m firebug loadtest=<test_file.py> <instance url>

Or use the internal windmill shell to explore available commands:

$ windmill -m firebug shell <instance url>

And enter python commands:

>>> load_test(<your test file>)
>>> run_test(<your test file>)

Integrate Windmill tests into CubicWeb#

Set environment#

You have to create a new unit test file and a windmill directory and copy all your windmill use case into it.

# test_windmill.py

# Run all scenarii found in windmill directory
from cubicweb.devtools.cwwindmill import (CubicWebWindmillUseCase,
                                          unittest_main)

if __name__ == '__main__':
    unittest_main()

Run your tests#

You can easily run your windmill test suite through pytest or unittest. You have to copy a test_windmill.py file from web.test.

To run your test series:

$ pytest test/test_windmill.py

By default, CubicWeb will use firefox as the default browser and will try to run test instance server on localhost. In the general case, You’ve no need to change anything.

Check cubicweb.devtools.cwwindmill.CubicWebWindmillUseCase for Windmill configuration. You can edit windmill settings with following class attributes:

  • browser identification string (firefox|ie|safari|chrome) (firefox by default)

  • test_dir testing file path or directory (windmill directory under your unit case file by default)

  • edit_test load and edit test for debugging (False by default)

Examples:

browser = 'firefox'
test_dir = osp.join(__file__, 'windmill')
edit_test = False

If you want to change cubicweb test server parameters, you can check class variables from CubicWebServerConfig or inherit it with overriding the configcls attribute in CubicWebServerTC

.. sourcecode:: python
class OtherCubicWebServerConfig(CubicWebServerConfig):

port = 9999

class NewCubicWebServerTC(CubicWebServerTC):

configcls = OtherCubicWebServerConfig

For instance, CubicWeb framework windmill tests can be manually run by:

$ pytest web/test/test_windmill.py

Edit your tests#

You can toggle the edit_test variable to enable test edition.

But if you are using pytest as test runner, use the -i option directly. The test series will be loaded and you can run assertions step-by-step:

$ pytest -i test/test_windmill.py

In this case, the firebug extension will be loaded automatically for you.

Afterwards, don’t forget to save your edited test into the right file (no autosave feature).

Best practises#

Don’t run another instance on the same port. You risk to silence some regressions (test runner will automatically fail in further versions).

Start your use case by using an assert on the expected primary url page. Otherwise all your tests could fail without clear explanation of the used navigation.

In the same location of the test_windmill.py, create a windmill/ with your windmill recorded use cases.

Caveats#

File Upload#

Windmill can’t do file uploads. This is a limitation of browser Javascript support / sandboxing, not of Windmill per se. It would be nice if there were some command that would prime the Windmill HTTP proxy to add a particular file to the next HTTP request that comes through, so that uploads could at least be faked.

Preferences#

A .windmill/prefs.py could be used to redefine default configuration values.

For managing browser extensions, read advanced topic chapter.

More configuration examples could be seen in windmill/conf/global_settings.py as template.

Writing text reports with RestructuredText#

CubicWeb offers several text formats for the RichString type used in schemas, including restructuredtext.

Three additional restructuredtext roles are defined by CubicWeb:

Importing relational data into a CubicWeb instance#

Introduction#

This tutorial explains how to import data from an external source (e.g. a collection of files) into a CubicWeb cube instance.

First, once we know the format of the data we wish to import, we devise a data model, that is, a CubicWeb (Yams) schema which reflects the way the data is structured. This schema is implemented in the schema.py file. In this tutorial, we will describe such a schema for a particular data set, the Diseasome data (see below).

Once the schema is defined, we create a cube and an instance. The cube is a specification of an application, whereas an instance is the application per se.

Once the schema is defined and the instance is created, the import can be performed, via the following steps:

  1. Build a custom parser for the data to be imported. Thus, one obtains a Python memory representation of the data.

  2. Map the parsed data to the data model defined in schema.py.

  3. Perform the actual import of the data. This comes down to “populating” the data model with the memory representation obtained at 1, according to the mapping defined at 2.

This tutorial illustrates all the above steps in the context of relational data stored in the RDF format.

More specifically, we describe the import of Diseasome RDF/OWL data.

Building a data model#

The first thing to do when using CubicWeb for creating an application from scratch is to devise a data model, that is, a relational representation of the problem to be modeled or of the structure of the data to be imported.

In such a schema, we define an entity type (EntityType objects) for each type of entity to import. Each such type has several attributes. If the attributes are of known CubicWeb (Yams) types, viz. numbers, strings or characters, then they are defined as attributes, as e.g. attribute = Int() for an attribute named attribute which is an integer.

Each such type also has a set of relations, which are defined like the attributes, except that they represent, in fact, relations between the entities of the type under discussion and the objects of a type which is specified in the relation definition.

For example, for the Diseasome data, we have two types of entities, genes and diseases. Thus, we create two classes which inherit from EntityType:

class Disease(EntityType):
    # Corresponds to http://www.w3.org/2000/01/rdf-schema#label
    label = String(maxsize=512, fulltextindexed=True)
    ...

    #Corresponds to http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseasome/associatedGene
    associated_genes = SubjectRelation('Gene', cardinality='**')
    ...

    #Corresponds to 'http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseasome/chromosomalLocation'
    chromosomal_location = SubjectRelation('ExternalUri', cardinality='?*', inlined=True)


class Gene(EntityType):
    ...

In this schema, there are attributes whose values are numbers or strings. Thus, they are defined by using the CubicWeb / Yams primitive types, e.g., label = String(maxsize=12). These types can have several constraints or attributes, such as maxsize. There are also relations, either between the entity types themselves, or between them and a CubicWeb type, ExternalUri. The latter defines a class of URI objects in CubicWeb. For instance, the chromosomal_location attribute is a relation between a Disease entity and an ExternalUri entity. The relation is marked by the CubicWeb / Yams SubjectRelation method. The latter can have several optional keyword arguments, such as cardinality which specifies the number of subjects and objects related by the relation type specified. For example, the '?*' cardinality in the chromosomal_relation relation type says that zero or more Disease entities are related to zero or one ExternalUri entities. In other words, a Disease entity is related to at most one ExternalUri entity via the chromosomal_location relation type, and that we can have zero or more Disease entities in the data base. For a relation between the entity types themselves, the associated_genes between a Disease entity and a Gene entity is defined, so that any number of Gene entities can be associated to a Disease, and there can be any number of Disease s if a Gene exists.

Of course, before being able to use the CubicWeb / Yams built-in objects, we need to import them:

from yams.buildobjs import EntityType, SubjectRelation, String, Int
from cubicweb.schemas.base import ExternalUri

Building a custom data parser#

The data we wish to import is structured in the RDF format, as a text file containing a set of lines. On each line, there are three fields. The first two fields are URIs (“Universal Resource Identifiers”). The third field is either an URI or a string. Each field bares a particular meaning:

  • the leftmost field is an URI that holds the entity to be imported. Note that the entities defined in the data model (i.e., in schema.py) should correspond to the entities whose URIs are specified in the import file.

  • the middle field is an URI that holds a relation whose subject is the entity defined by the leftmost field. Note that this should also correspond to the definitions in the data model.

  • the rightmost field is either an URI or a string. When this field is an URI, it gives the object of the relation defined by the middle field. When the rightmost field is a string, the middle field is interpreted as an attribute of the subject (introduced by the leftmost field) and the rightmost field is interpreted as the value of the attribute.

Note however that some attributes (i.e. relations whose objects are strings) have their objects defined as strings followed by ^^ and by another URI; we ignore this part.

Let us show some examples:

  • of line holding an attribute definition: <http://www4.wiwiss.fu-berlin.de/diseasome/resource/genes/CYP17A1> <http://www.w3.org/2000/01/rdf-schema#label> "CYP17A1" . The line contains the definition of the label attribute of an entity of type gene. The value of label is ‘CYP17A1’.

  • of line holding a relation definition: <http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseases/1> <http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseasome/associatedGene> <http://www4.wiwiss.fu-berlin.de/diseasome/resource/genes/HADH2> . The line contains the definition of the associatedGene relation between a disease subject entity identified by 1 and a gene object entity defined by HADH2.

Thus, for parsing the data, we can (:note: see the diseasome_parser module):

  1. define a couple of regular expressions for parsing the two kinds of lines, RE_ATTS for parsing the attribute definitions, and RE_RELS for parsing the relation definitions.

  2. define a function that iterates through the lines of the file and retrieves (yield s) a (subject, relation, object) tuple for each line. We called it _retrieve_structure in the diseasome_parser module. The function needs the file name and the types for which information should be retrieved.

Alternatively, instead of hand-making the parser, one could use the RDF parser provided in the dataio cube.

Once we get to have the (subject, relation, object) triples, we need to map them into the data model.

Mapping the data to the schema#

In the case of diseasome data, we can just define two dictionaries for mapping the names of the relations as extracted by the parser, to the names of the relations as defined in the schema.py data model. In the diseasome_parser module they are called MAPPING_ATTS and MAPPING_RELS. Given that the relation and attribute names are given in CamelCase in the original data, mappings are necessary if we follow the PEP08 when naming the attributes in the data model. For example, the RDF relation chromosomalLocation is mapped into the schema relation chromosomal_location.

Once these mappings have been defined, we just iterate over the (subject, relation, object) tuples provided by the parser and we extract the entities, with their attributes and relations. For each entity, we thus have a dictionary with two keys, attributes and relations. The value associated to the attributes key is a dictionary containing (attribute: value) pairs, where “value” is a string, plus the cwuri key / attribute holding the URI of the entity itself. The value associated to the relations key is a dictionary containing (relation: value) pairs, where “value” is an URI. This is implemented in the entities_from_rdf interface function of the module diseasome_parser. This function provides an iterator on the dictionaries containing the attributes and relations keys for all entities.

However, this is a simple case. In real life, things can get much more complicated, and the mapping can be far from trivial, especially when several data sources (which can follow different formatting and even structuring conventions) must be mapped into the same data model.

Importing the data#

The data import code should be placed in a Python module. Let us call it diseasome_import.py. Then, this module should be called via cubicweb-ctl, as follows:

cubicweb-ctl shell diseasome_import.py -- <other arguments e.g. data file>

In the import module, we should use a store for doing the import. A store is an object which provides three kinds of methods for importing data:

  • a method for importing the entities, along with the values of their attributes.

  • a method for importing the relations between the entities.

  • a method for committing the imports to the database.

In CubicWeb, we have four stores:

  1. ObjectStore base class for the stores in CubicWeb. It only provides a skeleton for all other stores and provides the means for creating the memory structures (dictionaries) that hold the entities and the relations between them.

  2. RQLObjectStore: store which uses the RQL language for performing database insertions and updates. It relies on all the CubicWeb hooks machinery, especially for dealing with security issues (database access permissions).

  1. NoHookRQLObjectStore: store which uses the RQL language for performing database insertions and updates, but for which all hooks are deactivated. This implies that certain checks with respect to the CubicWeb / Yams schema (data model) are not performed. However, all SQL queries obtained from the RQL ones are executed in a sequential manner, one query per inserted entity.

  1. SQLGenObjectStore: store which uses the SQL language directly. It inserts entities either sequentially, by executing an SQL query for each entity, or directly by using one PostGRES COPY FROM query for a set of similarly structured entities.

For really massive imports (millions or billions of entities), there is a cube dataio which contains another store, called MassiveObjectStore. This store is similar to SQLGenObjectStore, except that anything related to CubicWeb is bypassed. That is, even the CubicWeb EID entity identifiers are not handled. This store is the fastest, but has a slightly different API from the other four stores mentioned above. Moreover, it has an important limitation, in that it doesn’t insert inlined 1 relations in the database.

1

An inlined relation is a relation defined in the schema with the keyword argument inlined=True. Such a relation is inserted in the database as an attribute of the entity whose subject it is.

In the following section we will see how to import data by using the stores in CubicWeb’s dataimport module.

Using the stores in dataimport#

ObjectStore is seldom used in real life for importing data, since it is only the base store for the other stores and it doesn’t perform an actual import of the data. Nevertheless, the other three stores, which import data, are based on ObjectStore and provide the same API.

All three stores RQLObjectStore, NoHookRQLObjectStore and SQLGenObjectStore provide exactly the same API for importing data, that is entities and relations, in an SQL database.

Before using a store, one must import the dataimport module and then initialize the store, with the current session as a parameter:

import cubicweb.dataimport as cwdi
...

store = cwdi.RQLObjectStore(session)

Each such store provides three methods for data import:

  1. create_entity(Etype, **attributes), which allows us to add an entity of the Yams type Etype to the database. This entity’s attributes are specified in the attributes dictionary. The method returns the entity created in the database. For example, we add two entities, a person, of Person type, and a location, of Location type:

    person = store.create_entity('Person', name='Toto', age='18', height='190')

location = store.create_entity('Location', town='Paris', arrondissement='13')

  2. relate(subject_eid, r_type, object_eid), which allows us to add a relation of the Yams type r_type to the database. The relation’s subject is an entity whose EID is subject_eid; its object is another entity, whose EID is object_eid. For example 2:

    store.relate(person.eid(), 'lives_in', location.eid(), **kwargs)

    kwargs is only used by the SQLGenObjectStore’s relate method and is here to allow us to specify the type of the subject of the relation, when the relation is defined as inlined in the schema.

2
The eid method of an entity defined via create_entity returns

the entity identifier as assigned by CubicWeb when creating the entity. This only works for entities defined via the stores in the CubicWeb’s dataimport module.

The keyword argument that is understood by SQLGenObjectStore is called subjtype and holds the type of the subject entity. For the example considered here, this comes to having 3:

store.relate(person.eid(), 'lives_in', location.eid(), subjtype=person.cw_etype)

If subjtype is not specified, then the store tries to infer the type of the subject. However, this doesn’t always work, e.g. when there are several possible subject types for a given relation type.

3
The cw_etype attribute of an entity defined via create_entity holds

the type of the entity just created. This only works for entities defined via the stores in the CubicWeb’s dataimport module. In the example considered here, person.cw_etype holds 'Person'.

All the other stores but SQLGenObjectStore ignore the kwargs parameters.

  1. flush(), which allows us to perform the actual commit into the database, along with some cleanup operations. Ideally, this method should be called as often as possible, that is after each insertion in the database, so that database sessions are kept as atomic as possible. In practice, we usually call this method twice: first, after all the entities have been created, second, after all relations have been created.

    Note however that before each commit the database insertions have to be consistent with the schema. Thus, if, for instance, an entity has an attribute defined through a relation (viz. a SubjectRelation) with a "1" or "+" object cardinality, we have to create the entity under discussion, the object entity of the relation under discussion, and the relation itself, before committing the additions to the database.

    The flush method is simply called as:

    store.flush().

Using the MassiveObjectStore in the dataio cube#

This store, available in the dataio cube, allows us to fully dispense with the CubicWeb import mechanisms and hence to interact directly with the database server, via SQL queries.

Moreover, these queries rely on PostGreSQL’s COPY FROM instruction to create several entities in a single query. This brings tremendous performance improvements with respect to the RQL-based data insertion procedures.

However, the API of this store is slightly different from the API of the stores in CubicWeb’s dataimport module.

Before using the store, one has to import the dataio cube’s dataimport module, then initialize the store by giving it the session parameter:

from cubicweb_dataio import dataimport as mcwdi
...

store = mcwdi.MassiveObjectStore(session)

The MassiveObjectStore provides six methods for inserting data into the database:

  1. init_rtype_table(SubjEtype, r_type, ObjEtype), which specifies the creation of the tables associated to the relation types in the database. Each such table has three column, the type of the subject entity, the type of the relation (that is, the name of the attribute in the subject entity which is defined via the relation), and the type of the object entity. For example:

    store.init_rtype_table('Person', 'lives_in', 'Location')

    Please note that these tables can be created before the entities, since they only specify their types, not their unique identifiers.

  2. create_entity(Etype, **attributes), which allows us to add new entities, whose attributes are given in the attributes dictionary. Please note however that, by default, this method does not return the created entity. The method is called, for example, as in:

    store.create_entity('Person', name='Toto', age='18', height='190',
                    uri='http://link/to/person/toto_18_190')
store.create_entity('Location', town='Paris', arrondissement='13',
                    uri='http://link/to/location/paris_13')

    In order to be able to link these entities via the relations when needed, we must provide ourselves a means for uniquely identifying the entities. In general, this is done via URIs, stored in attributes like uri or cwuri. The name of the attribute is irrelevant as long as its value is unique for each entity.

  3. relate_by_iid(subject_iid, r_type, object_iid) allows us to actually relate the entities uniquely identified by subject_iid and object_iid via a relation of type r_type. For example:

    store.relate_by_iid('http://link/to/person/toto_18_190',
                    'lives_in',
                    'http://link/to/location/paris_13')

    Please note that this method does not work for inlined relations!

  4. convert_relations(SubjEtype, r_type, ObjEtype, subj_iid_attribute, obj_iid_attribute) allows us to actually insert the relations in the database. At one call of this method, one inserts all the relations of type rtype between entities of given types. subj_iid_attribute and object_iid_attribute are the names of the attributes which store the unique identifiers of the entities, as assigned by the user. These names can be identical, as long as their values are unique. For example, for inserting all relations of type lives_in between People and Location entities, we write:

    store.convert_relations('Person', 'lives_in', 'Location', 'uri', 'uri')

  5. flush() performs the actual commit in the database. It only needs to be called after create_entity and relate_by_iid calls. Please note that relate_by_iid does not perform insertions into the database, hence calling flush() for it would have no effect.

  6. cleanup() performs database cleanups, by removing temporary tables. It should only be called at the end of the import.

Application to the Diseasome data#

Import setup#

We define an import function, diseasome_import, which does basically four things:

  1. creates and initializes the store to be used, via a line such as:

    store = cwdi.SQLGenObjectStore(session)

    where cwdi is the imported cubicweb.dataimport or cubicweb_dataio.dataimport.

  2. calls the diseasome parser, that is, the entities_from_rdf function in the diseasome_parser module and iterates on its result, in a line such as:

    for entity, relations in parser.entities_from_rdf(filename, ('gene', 'disease')):

    where parser is the imported diseasome_parser module, and filename is the name of the file containing the data (with its path), e.g. ../data/diseasome_dump.nt.

  3. creates the entities to be inserted in the database; for Diseasome, there are two kinds of entities:

    1. entities defined in the data model, viz. Gene and Disease in our case.

    2. entities which are built in CubicWeb / Yams, viz. ExternalUri which define URIs.

    As we are working with RDF data, each entity is defined through a series of URIs. Hence, each “relational attribute” 4 of an entity is defined via an URI, that is, in CubicWeb terms, via an ExternalUri entity. The entities are created, in the loop presented above, as such:

    ent = store.create_entity(etype, **entity)

    where etype is the appropriate entity type, either Gene or Disease.

4
By “relational attribute” we denote an attribute (of an entity) which

is defined through a relation, e.g. the chromosomal_location attribute of Disease entities, which is defined through a relation between a Disease and an ExternalUri.

The ExternalUri entities are as many as URIs in the data file. For them, we define a unique attribute, uri, which holds the URI under discussion:

extu = store.create_entity('ExternalUri', uri="http://path/of/the/uri")

  1. creates the relations between the entities. We have relations between:

    1. entities defined in the schema, e.g. between Disease and Gene entities, such as the associated_genes relation defined for Disease entities.

    2. entities defined in the schema and ExternalUri entities, such as gene_id.

    The way relations are added to the database depends on the store:

    • for the stores in the CubicWeb dataimport module, we only use store.relate, in another loop, on the relations (that is, a loop inside the preceding one, mentioned at step 2):

      for rtype, rels in relations.iteritems():
    ...

    store.relate(ent.eid(), rtype, extu.eid(), **kwargs)

      where kwargs is a dictionary designed to accommodate the need for specifying the type of the subject entity of the relation, when the relation is inlined and SQLGenObjectStore is used. For example:

      ...
store.relate(ent.eid(), 'chromosomal_location', extu.eid(), subjtype='Disease')

    • for the MassiveObjectStore in the dataio cube’s dataimport module, the relations are created in three steps:

      1. first, a table is created for each relation type, as in:

        ...
store.init_rtype_table(ent.cw_etype, rtype, extu.cw_etype)

        which comes down to lines such as:

        store.init_rtype_table('Disease', 'associated_genes', 'Gene')
store.init_rtype_table('Gene', 'gene_id', 'ExternalUri')

      2. second, the URI of each entity will be used as its identifier, in the relate_by_iid method, such as:

        disease_uri = 'http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseases/3'
gene_uri = '<http://www4.wiwiss.fu-berlin.de/diseasome/resource/genes/HSD3B2'
store.relate_by_iid(disease_uri, 'associated_genes', gene_uri)

      3. third, the relations for each relation type will be added to the database, via the convert_relations method, such as in:

        store.convert_relations('Disease', 'associated_genes', 'Gene', 'cwuri', 'cwuri')

        and:

        store.convert_relations('Gene', 'hgnc_id', 'ExternalUri', 'cwuri', 'uri')

        where cwuri and uri are the attributes which store the URIs of the entities defined in the data model, and of the ExternalUri entities, respectively.

  2. flushes all relations and entities:

    store.flush()

    which performs the actual commit of the inserted entities and relations in the database.

If the MassiveObjectStore is used, then a cleanup of temporary SQL tables should be performed at the end of the import:

store.cleanup()
Timing benchmarks#

In order to time the import script, we just decorate the import function with the timed decorator:

from logilab.common.decorators import timed
...

@timed
def diseasome_import(session, filename):
    ...

After running the import function as shown in the “Importing the data” section, we obtain two time measurements:

diseasome_import clock: ... / time: ...

Here, the meanings of these measurements are 5:

  • clock is the time spent by CubicWeb, on the server side (i.e. hooks and data pre- / post-processing on SQL queries),

  • time is the sum between clock and the time spent in PostGreSQL.

5

The meanings of the clock and time measurements, when using the @timed decorators, were taken from a blog post on massive data import in CubicWeb.

The import function is put in an import module, named diseasome_import here. The module is called directly from the CubicWeb shell, as follows:

cubicweb-ctl shell diseasome_instance diseasome_import.py \
-- -df diseasome_import_file.nt -st StoreName

The module accepts two arguments:

  • the data file, introduced by -df [--datafile], and

  • the store, introduced by -st [--store].

The timings (in seconds) for different stores are given in the following table, for importing 4213 Disease entities and 3919 Gene entities with the import module just described:

Store

CubicWeb time (clock)

PostGreSQL time (time - clock)

Total time

RQLObjectStore

225.98

62.05

288.03

NoHookRQLObjectStore

62.73

51.38

114.11

SQLGenObjectStore

20.41

11.03

31.44

MassiveObjectStore

4.84

6.93

11.77

Conclusions#

In this tutorial we have seen how to import data in a CubicWeb application instance. We have first seen how to create a schema, then how to create a parser of the data and a mapping of the data to the schema. Finally, we have seen four ways of importing data into CubicWeb.

Three of those are integrated into CubicWeb, namely the RQLObjectStore, NoHookRQLObjectStore and SQLGenObjectStore stores, which have a common API:

  • RQLObjectStore is by far the slowest, especially its time spent on the CubicWeb side, and so it should be used only for small amounts of “sensitive” data (i.e. where security is a concern).

  • NoHookRQLObjectStore slashes by almost four the time spent on the CubicWeb side, but is also quite slow; on the PostGres side it is as slow as the previous store. It should be used for data where security is not a concern, but consistency (with the data model) is.

  • SQLGenObjectStore slashes by three the time spent on the CubicWeb side and by five the time spent on the PostGreSQL side. It should be used for relatively great amounts of data, where security and data consistency are not a concern. Compared to the previous store, it has the disadvantage that, for inlined relations, we must specify their subjects’ types.

For really huge amounts of data there is a fourth store, MassiveObjectStore, available from the dataio cube. It provides a blazing performance with respect to all other stores: it is almost 25 times faster than RQLObjectStore and almost three times faster than SQLGenObjectStore. However, it has a few usage caveats that should be taken into account:

  1. it cannot insert relations defined as inlined in the schema,

  2. no security or consistency check is performed on the data,

  3. its API is slightly different from the other stores.

Hence, this store should be used when security and data consistency are not a concern, and there are no inlined relations in the schema.

Create a Website from scratch with CubicWeb#

Introduction#

This tutorial aims to demonstrate how to create a website using CubicWeb. This website will present museums from French Ministry of Culture data, available here.

First, we will start with installation and creation of our website, and a short presentation of out of the box CubicWeb functionalities. Then, we will see how to enhance our views using Jinja2 templates or React components to have a better looking site. Finally, we will see how to manage more data, and how to serialize them in RDF.

At the end of this tutorial, you will have a website giving information about all France’s museums, describes them in RDF and present them on a map.

You can find the code of the finished tutorial in our forge, look for the cube tuto.

Getting started#

Installation of CubicWeb and dependencies#

In this tutorial, we choose to install CubicWeb as a Python Package in a Python3 virtual environment, with pip; instead of using Debian installation. We will also need to install psycopg2-binary if we use a postgresql database:

python3 -m venv venv-tuto
source venv-tuto/bin/activate
pip install cubicweb
pip install psycopg2-binary
Create a cube#

Now we have CubicWeb installed, we will need to create a cube, which will contain our application. We will call our cube tuto:

cubicweb-ctl newcube tuto -d tuto

This command will lead to several questions, as a short description of the new cube. Then, it will create a directory named tuto (as we specified it with -d) reflecting the structure described in Standard structure for a cube.

To install our new cube on the virtual environment, run in the tuto/cubicweb-tuto directory:

pip install -e .

All cubicweb-ctl commands are described in details in cubicweb-ctl tool.

Create and start our instance#

Now we created our cube, we need to instantiate it to launch our website:

cubicweb-ctl create tuto tuto_instance

Several questions will be asked to parameter our new instance, most of them can be answered with default value, some (as DB user and password) can’t. At the end, it will be asked if we want to create the database now. We do not need it right now, as we will create in further steps.

Then, we can launch our instance in debug mod (with -D option):

cubicweb-ctl pyramid -D tuto_instance

You can now access the instance from http://localhost:8080

The empty instance homepage.

As you can see, we already have several functionalities which come out-of-the-box, for instance user management, data model schema displaying, etc.

Now, we need to design our data model, to be able to create and display some museums.

Defining our data model#

We want to display some museums, each have a name, a postal address, maybe one or several director, a geographical position (latitude and longitude) and are in a city. Some of these concepts will be classes, others attributes.

Thus, we will write the following code in our tuto/cubicweb_tuto/schema.py file:

from yams.buildobjs import EntityType, String, Float, RelationDefinition, Int


class Museum(EntityType):
    name = String()
    latitude = Float()
    longitude = Float()
    postal_address = String()


class City(EntityType):
    name = String()
    zip_code = Int()


class Person(EntityType):
    name = String()
    email = String()

class is_in(RelationDefinition):
    subject = 'Museum'
    object = 'City'
    cardinality = '1*'

class director(RelationDefinition):
    subject = 'Museum'
    object = 'Person'
    cardinality = '**'

The first step is the import from the yams package necessary classes to build the schema.

This file defines the following:

  • a Museum has a name, a latitude, a longitude and a postal address as attributes.

    • the name and postal address are strings;

    • the latitude and longitude are floating numbers.

  • a City has a name and a zip code as attributes.

  • a Person has a name and an email as attributes

  • a Museum must be linked to a City using the is_in relation

    • * means a City may be linked to 0 to N Museum, 1 means a Museum must be linked to one and only one City. For completeness, you can also use + for 1 to N, and ? for 0 or 1.

  • a Museum can be linked to 0 or several Person using the director relation, and a Person can be linked to 0 or several Museum.

Of course, there are a lot of other data types and things such as constraints, permissions, etc, that may be defined in the schema, but those will not be covered in this tutorial.

In our case, our relations have only on subject type. Thus, we can define them directly in Museum class, using SubjectRelation, like this:

from yams.buildobjs import EntityType, String, Float, SubjectRelation, Int


class Museum(EntityType):
    name = String()
    latitude = Float()
    longitude = Float()
    is_in = SubjectRelation("City", cardinality="1*")
    director = SubjectRelation("Person", cardinality="**")
    postal_address = String()


class City(EntityType):
    name = String()
    zip_code = Int()


class Person(EntityType):
    name = String()
    email = String()

Once the schema created, we need to create our database, and then initialise it with the aforementioned schema:

cubicweb-ctl db-create tuto_instance
cubicweb-ctl db-init tuto_instance

Note

At the end of the db-create, it is asked if we want to run database initialisation, thus we can avoid running db-init.

Note

In our case, we had no existing database, so we had to initialize a database. But with a pre-existing database and schema, we have to use migration scripts, see Migration for more information about this topic.

If we launch again our instance, we should see our new entity types in the homepage: City, Museum, Person; and for each, the number of instance of these types (currently 0, as we don’t have any of these entities).

The instance homepage, with new entity types.

By clicking on data model schema, we can see our data model, with our three classes and two relations.

The instance data model schema, with new entity types.
Adding data#

Now we have our entity types defined, we will see how to add some entities. To do this, we need to be connected as administrator, using the login button at right top of the site, or visiting http://localhost:8080/login. As you can see, we have more choices in the homepage, and beside each entity type, we have a +, allowing to create a new entity of this type.

The instance homepage, in administrator mode.

As we built our schema, a Museum have to be linked to a City, so we first need to create a City before adding a museum. To do this, we just have to click on the + beside City (0), and fill the form.

City entity creation.

As you can see, all the fields comes directly from the schema and the form is automatically generated by CubicWeb.

When all the fields are fulfilled, we just have to validate, and we are redirected on the city page, where we can see its different attributes, and on the left, several possible action; as modify our entity or delete it.

City entity view.

Now we have our first city, we will add its three museums. As for the city creation, we have an autogenerated form; but with a little particularity: a field to choose the city to link with our museum. This field must be fulfilled to create our entity.

Museum entity creation.

As for the city, we are redirected on the entity view after its creation.

Museum entity view.

We then add two other museums. When we go back to the homepage, we can see all three museums when we click on Museum_plural (3).

Our three museums.

If we click on City in the homepage, we do not have a list view, but our single entity view. This is because in the first case, the framework chose to use the ‘primary’ view since there is only one entity in the data to be displayed. As we have three museums, the ‘list’ view is more appropriate and hence is being used.

There are various other places where CubicWeb adapts to display data in the best way, the main being provided by the view selection mechanism that will be detailed later.

Customize museum primary view#

The ‘primary’ view (i.e. any view with the identifier set to ‘primary’) is the one used to display all the information about a single entity. The standard primary view is one of the most sophisticated views of all. It has several customisation points, but its power comes with uicfg, allowing you to control it without having to subclass it. More information are available here : The Primary View.

Now we have several museums, we want an easier way to identify its city when we are on the museum page. To achieve this, we will subclass PrimaryView and override render_entity_title method in tuto/cubicweb_tuto/views.py:

from cubicweb.predicates import is_instance
from cubicweb.web.views.primary import PrimaryView


class MuseumPrimaryView(PrimaryView):
    __select__ = is_instance("Museum")

    def render_entity_title(self, entity):
        """Renders the entity title.
        """
        city_name = entity.is_in[0].name
        self.w(f"<h1>{entity.name} ({city_name})</h1>")

As stated before, CubicWeb comes with a system of views selection. This system is, among other things, based on selectors declared with __select__ (you’ll find more information about this in the Registries and application objects chapter). As we want to customize museum primary view, we use __select__ = is_instance(“Museum”) to tell CubicWeb this is only applicable when we display a Museum entity.

Then, we just override the method used to compute title to add the city name. To reach the city name, we use the relation is_in and choose the first and only one linked city, then ask for its name.

Museum entity customized with city name view.
Use entities.py to add more logic#

CubicWeb provides an ORM to easily programmaticaly manipulate entities. By default, entity types are instances of the AnyEntity class, which holds a set of predefined methods as well as property automatically generated for attributes/relations of the type it represents.

You can redefine each entity to provide additional methods or whatever you want to help you write your application. Customizing an entity requires that your entity:

  • inherits from cubicweb.entities.AnyEntity or any subclass

  • defines a __regid__ linked to the corresponding data type of your schema

You may then want to add your own methods, override default implementation of some method, etc…

As we may want reuse our custom museum title (with city name, as defined in previous section), we will define it as a property of our Museum class.

To do so, write this code in tuto/cubicweb_tuto/entities.py:

from cubicweb.entities import AnyEntity, fetch_config


class Museum(AnyEntity):
    __regid__ = "Museum"

    @property
    def title_with_city(self):
        return f"{self.name} ({self.is_in[0].name})"

Then, we just have to use it our previously defined view in tuto/cubicweb_tuto/views.py:

from cubicweb.predicates import is_instance
from cubicweb.web.views.primary import PrimaryView


class MuseumPrimaryView(PrimaryView):
    __select__ = is_instance("Museum")

    def render_entity_title(self, entity):
        """Renders the entity title.
        """
        self.w(f"<h1>{entity.title_with_city}</h1>")
Conclusion#

In this first part, we laid the cornerstone of our futur site, and discovered some core functionalities of CubicWeb. In next parts, we will improve views and see how to import all our data.

Enhance views#

In Getting started, we saw how to develop our views by writing html code directly in CubicWeb views. In this part, we will see how to customize our web application using different methods : with pyramid views using jinja2 templates and with React.

Pyramid and Jinja2#
React in a CubicWeb view#

In this section, we want to add a map in museum pages to display where is the museum associated with the page.

To do this, we will use React simple maps, a React library. Our goal is to add a react component inside our museum primary view.

First, we will setup our environment. At logilab, we use Typescript when it is possible, so we will use it also in this tutorial. As module builder, we will use Webpack.

Thus, we need to create three files at the root of our cube: package.json, tsconfig.json and webpack.config.js. A lot of documentation can be find on the Web about how to configure a React/Typescript environment, so we are not going to dwell on it in this tutorial; and we will simply copy and paste the following files.

package.json:

{
  "name": "cubicweb_tuto",
  "version": "1.0.0",
  "description": "Summary ------- A cube for new CW tutorial",
  "directories": {
    "test": "test"
  },
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "webpack",
    "watch": "webpack --watch --mode=development"
  },
  "author": "Logilab",
  "license": "GPL-2.0-or-later",
  "dependencies": {
    "@types/react": "^17.0.0",
    "@types/react-dom": "^17.0.0",
    "@types/react-simple-maps": "^1.0.3",
    "prop-types": "^15.7.2",
    "react": "^17.0.1",
    "react-dom": "^17.0.1",
    "react-simple-maps": "^2.3.0",
    "ts-loader": "^8.0.14",
    "typescript": "^4.1.3",
    "webpack": "^5.18.0",
    "webpack-cli": "^4.4.0"
  }
}

tsconfig.json:

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "jsx": "react",
    "strict": true,
    "esModuleInterop": true
  }
}

webpack.config.js:

const path = require("path");

module.exports = {
  entry: {
    "map.js": "./appjs/geomap.tsx",
  },
  output: {
    filename: "[name]",
    path: path.resolve(__dirname, "./cubicweb_tuto/data/")
  },
  resolve: {
    extensions: [".tsx", ".ts", ".jsx", ".js"]
  },
  module: {
    rules: [
      {
        test: [/\.tsx?$/],
        exclude: /node_modules/,
        use: ["ts-loader"]
      }
    ]
  },
  plugins: []
};

Now we have our configuration files, we have to install NodeJS and then install our project using npm.

sudo apt-get install nodejs
npm install

They are two last things to do:

  • create a component to display a museum on the map;

  • integrate our component in a CubicWeb view.

By convention, we put our js files in a appjs directory, and bundle are built in cubicweb_tuto/data (as you can see in our webpack.config.js). Then, we will create a file geomap.tsx in appjs/.

For our component, we will need three parameters: our museum name, its latitude and its longitude. These parameters will be defined in our CubicWeb view when we will call our script. Our file geomap.tsx can be written like this:

import React from 'react';
import ReactDOM from 'react-dom';
import {
  ComposableMap,
  Geographies,
  Geography,
  Marker,
  Point
} from "react-simple-maps";

const geoUrl = "https://raw.githubusercontent.com/zcreativelabs/react-simple-maps/master/topojson-maps/world-110m.json";

declare const data: {
    name: string,
    latitude: number,
    longitude: number,
}

const MapChart = () => {
  return (
    <ComposableMap>
      <Geographies geography={geoUrl}>
        {({ geographies }) =>
          geographies
            .map(geo => (
              <Geography
                key={geo.rsmKey}
                geography={geo}
                fill="#EAEAEC"
                stroke="#D6D6DA"
              />
            ))
        }
      </Geographies>
        <Marker coordinates={[data.longitude, data.latitude] as Point}>
          <g
            fill="none"
            stroke="#FF5533"
            strokeWidth="2"
            strokeLinecap="round"
            strokeLinejoin="round"
            transform="translate(-12, -24)"
          >
            <circle cx="12" cy="10" r="3" />
            <path d="M12 21.7C17.3 17 20 13 20 10a8 8 0 1 0-16 0c0 3 2.7 6.9 8 11.7z" />
          </g>
          <text
            textAnchor="middle"
            y={10}
            style={{ fontFamily: "system-ui", fill: "#5D5A6D" }}
          >
            {data.name}
          </text>
        </Marker>
    </ComposableMap>
  );
};

function App() {
    return <MapChart/>
}

const root = document.getElementById("awesome-map");
ReactDOM.render(<App/>, root);

Now we will override the render_entity(self, entity) function of the Museum PrimaryView, in cubicweb-tuto/views.py to add:

  • the bundle javascript including our component;

  • a div with the id awesome-map which will be used by our component.

class MuseumPrimaryView(PrimaryView):
__select__ = is_instance("Museum")

def render_entity(self, entity):
    self.render_entity_toolbox(entity)
    self.render_entity_title(entity)
    # entity's attributes and relations, excluding meta data
    # if the entity isn't meta itself
    if self.is_primary():
        boxes = self._prepare_side_boxes(entity)
    else:
        boxes = None
    if boxes or hasattr(self, "render_side_related"):
        self.w('<table width="100%"><tr><td style="width: 75%">')

    self.w('<div class="mainInfo">')
    self.content_navigation_components("navcontenttop")
    self.render_entity_attributes(entity)
    if self.main_related_section:
        self.render_entity_relations(entity)
    self.render_map(entity)
    self.content_navigation_components("navcontentbottom")
    self.w("</div>")
    # side boxes
    if boxes or hasattr(self, "render_side_related"):
        self.w("</td><td>")
        self.w('<div class="primaryRight">')
        self.render_side_boxes(boxes)
        self.w("</div>")
        self.w("</td></tr></table>")

def render_entity_title(self, entity):
    """Renders the entity title, by default using entity's
    :meth:`dc_title()` method.
    """
    self.w(f"<h1>{entity.title_with_city}</h1>")

def render_map(self, entity):
    """Renders a map displaying where the museum is."""
    if not (entity.latitude and entity.longitude):
        return
    js_file = f"{self._cw.vreg.config.datadir_url}map.js"
    data = json_dumps(entity)
    self.w('<div id="awesome-map"></div>')
    self.w(
        f"""
        <script type="text/javascript">
            const data = {data};
        </script>
        <script src={js_file}></script>
    """
    )

Most part of render_entity(self, entity) are the same as its definition in PrimaryView, except that we add a call to render_map(self, entity); which will add a div tag with a specific id and a script tag adding our javascript bundle, and define variables containing information to display a museum on the map. The specific id must be the same as the one we defined in our javascript file, awesome-map.

Now, it’s time to build the javascript bundle using:

npm run build

And then, run our application:

cubicweb-ctl pyramid -D tuto_instance

We now have a world map displaying the location of our museum on museum pages. A lot of things could be done to have a better result, like center the map on the museum, but it’s out of the scope of this tutorial.

Our application with a World Map.
React in a Pyramid view#

Data management with CubicWeb#

Import data#

With our application customized, let’s see how to import more data. There is several ways to import data in CubicWeb. In our tutorial, we want to import our museums from a csv file. This file is provided by the France’s Ministry of Culture, and is available here.

There are several ways to import data in CubicWeb; in this tutorial, we will use one of them, the others are described here: Data Import.

First of all, we define in tuto/cubicweb_tuto/dataimport.py a function which will read a file from a filepath and create the corresponding entities, using a CubicWeb connection:

import csv


def import_museums(cnx, filepath):
    existing_cities = dict(cnx.execute("Any Z, C Where C is City, C zip_code Z"))
    existing_cities_nb = len(existing_cities)
    created_museum_nb = 0
    with open(filepath) as fileobj:
        reader = csv.DictReader(fileobj, delimiter=";")
        for record in reader:
            museum_name = record["NOM DU MUSEE"]
            street = record["ADR"]
            zip_code = record["CP"]
            city_name = record["VILLE"]
            try:
                lat, lng = record["coordonnees_finales"].split(",")
                lat_long = {
                    "latitude": lat,
                    "longitude": lng,
                }
            except (AttributeError, ValueError):
                lat_long = {}
            try:
                city = existing_cities[zip_code]
            except KeyError:
                city = cnx.create_entity("City", name=city_name, zip_code=zip_code)
                existing_cities[zip_code] = city.eid
            cnx.create_entity(
                "Museum",
                name=museum_name,
                postal_address=f"{street}, {zip_code} {city_name}",
                is_in=city,
                **lat_long,
            )
            created_museum_nb += 1

    print(
        "Import finished! {} existing cities, {} cities created, {} museums created.".format(
            existing_cities_nb,
            len(existing_cities) - existing_cities_nb,
            created_museum_nb,
        )
    )

To be sure we don’t have several time the same city, we first query CubicWeb to ask for all existing city. To do this, we use a specific language called RQL. As for SPARQL, it’s a query language designed to query linked data. See Introduction for more information about it.

Then, we put existing cities in a dictionary, using zip code as key. In the real world, a zip code can concern several cities, but it’s not really an issue in this tutorial.

For each line of our csv file, we will check if we already have the city in our base. If not, we create it. Then, we create our Museum entity with all needed arguments.

To create an entity, we use the create_entity method of the CubicWeb connection. This method takes as first argument the type of the entity (ie: the name of the corresponding class previously defined in tuto/cubicweb_tuto/schema.py), and then all arguments of the entity type.

In our example, a city needs a name and a zip code. A museum needs a name, a postal address, a latitude, a longitude and a city. As is_in is a relation, we give to the corresponding argument the eid of the city.

Note

As we have defined Museum in the schema, we have to link each instance of Museum to a City, that’s why we create the city before the museum, and give this city as argument of the museum.

If the city wasn’t mandatory, we could add it later, using:

museum_entity.cw_set(is_in=city)

To use our function we need to create a CubicWeb command that will call it. First, we create a file tuto/cubicweb_tuto/ccplugin.py (the name doesn’t matter, but it is commonly used for all new CubicWeb commands). Then, we write the following code:

from cubicweb.cwctl import CWCTL
from cubicweb.toolsutils import Command
from cubicweb.utils import admincnx

from cubicweb_tuto.dataimport import import_museums


@CWCTL.register
class ImportMuseums(Command):
    """
    Import Museums and Cities from a CSV from:
    https://data.culture.gouv.fr/explore/dataset/liste-et-localisation-des-musees-de-france/export/
    """

    arguments = "<instance> <csv_file>"
    name = "import-museums"
    min_args = max_args = 2

    def run(self, args):
        appid, csv_file = args[:2]

        with admincnx(appid) as cnx:
            import_museums(cnx, csv_file)
            cnx.commit()

  • @CWCTL.register allows to register the command and then make it available with cubicweb-ctl command by its name.

  • arguments defines which arguments take our command.

  • name defines the name of the command.

  • with admincnx(appid) as cnx allows to have an admin access to our instance, and then be able to create new entities.

Thus, to execute our import command, we just have to enter in our shell (within our virtual env):

cubicweb-ctl import-museums tuto_instance <path_to_the_csv>

After this script, we should be able to see that we have much more cities and museums by visiting the homepage of our CubicWeb instance:

A CubicWeb instance with several cities and museums.
RDF serialisation#
Content negotiation#

Setup and Administration#

This part is for installation and administration of the CubicWeb framework and instances based on that framework.

Install a CubicWeb environment#

Official releases are available from the CubicWeb.org forge and from PyPI. Since CubicWeb is developed using Agile software development techniques, releases happen frequently. In a version numbered X.Y.Z, X changes after a few years when the API breaks, Y changes after a few weeks when features are added and Z changes after a few days when bugs are fixed.

There are several ways to install CubicWeb depending on your needs:

If you are a power-user and need the very latest features, you can choose the following methods:

Additional configuration can be found in the section Configure a CubicWeb environment for better control and advanced features of CubicWeb.

Installing Dependencies#

No matter your installation method, you will need to install the following Debian packages:

apt install gettext graphviz

gettext is used for translations (see Internationalization), and graphviz to display relation schemas within the website.

Installing CubicWeb#

Docker install#

Detailed instructions on how to deploy CubicWeb using docker can be found on the docker hub.

Images are built using the source code available in the docker-cubicweb repository.

Virtualenv install#

CubicWeb can be safely installed, used and contained inside a virtualenv. To create and activate a virtualenv, use the following commands:

pip install --user virtualenv
virtualenv venv
source venv/bin/activate

Then you can use either pip or easy_install to install CubicWeb inside an activated virtual environment.

pip install#

pip is a python tool that helps downloading, building, installing, and managing Python packages and their dependencies. It is fully compatible with virtualenv and installs the packages from sources published on the The Python Package Index.

A working compilation chain is needed to build modules which include C extensions. If you really do not want to compile anything, installing lxml, and libgecode will help.

For Debian, these minimal dependencies can be obtained by doing:

apt install gcc python3-pip python3-dev python3-lxml

or, if you prefer to get as much as possible from pip:

apt install gcc python3-pip python3-dev libxslt1-dev libxml2-dev

For Windows, you can install pre-built packages (possible source). For a minimal setup, install:

Make sure to choose the correct architecture and version of Python.

Finally, install CubicWeb and its dependencies by running:

pip install cubicweb

Install from source#

You can download the archive containing the sources from CubicWeb forge downloads section.

Make sure you also have all the dependencies installed.

Once uncompressed, you can install the framework from inside the uncompressed folder with:

python3 setup.py install

Or you can run CubicWeb directly from the source directory by setting the resource mode to user. This will ease the development with the framework.

Install from version control system#

To keep-up with on-going development, clone the Mercurial repository:

hg clone -u 'last(tag())' https://forge.extranet.logilab.fr/cubicweb/cubicweb # stable version
hg clone https://forge.extranet.logilab.fr/cubicweb/cubicweb # development branch

Make sure you also have all the Installation dependencies.

Installing cubes#

Many other cubes are available. Those cubes can help expanding the functionalities offered by CubicWeb. A list is available at PyPI or at the CubicWeb.org forge.

For example the blog cube can be installed using:

pip install cubicweb-blog

Configure a CubicWeb environment#

You can configure the database system of your choice:

For advanced features, have a look to:

Databases configuration#

Each instance can be configured with its own database connection information, that will be stored in the instance’s sources file. The database to use will be chosen when creating the instance. CubicWeb is known to run with Postgresql (recommended) and SQLite.

Other possible sources of data include CubicWeb, LDAP and Mercurial, but at least one relational database is required for CubicWeb to work. You do not need to install a backend that you do not intend to use for one of your instances. SQLite is not fit for production use, but it works well for testing and ships with Python, which saves installation time when you want to get started quickly.

PostgreSQL#

Many Linux distributions ship with the appropriate PostgreSQL packages. Basically, you need to install the following packages:

  • postgresql and postgresql-client, which will pull the respective versioned packages (e.g. postgresql-9.1 and postgresql-client-9.1) and, optionally,

  • a postgresql-plpython-X.Y package with a version corresponding to that of the aforementioned packages (e.g. postgresql-plpython-9.1). (Not needed now by default)

If you run postgres on another host than the CubicWeb repository, you should install the postgresql-client package on the CubicWeb host, and others on the database host.

For extra details concerning installation, please refer to the PostgreSQL project online documentation.

Database cluster#

If you already have an existing cluster and PostgreSQL server running, you do not need to execute the initilization step of your PostgreSQL database unless you want a specific cluster for CubicWeb databases or if your existing cluster doesn’t use the UTF8 encoding (see note below).

To initialize a PostgreSQL cluster, use the command initdb:

$ initdb -E UTF8 -D /path/to/pgsql

Note: initdb might not be in the PATH, so you may have to use its absolute path instead (usually something like /usr/lib/postgresql/9.4/bin/initdb).

Notice the encoding specification. This is necessary since CubicWeb usually want UTF8 encoded database. If you use a cluster with the wrong encoding, you’ll get error like:

new encoding (UTF8) is incompatible with the encoding of the template database (SQL_ASCII)
HINT:  Use the same encoding as in the template database, or use template0 as template.

Once initialized, start the database server PostgreSQL with the command:

$ postgres -D /path/to/psql

If you cannot execute this command due to permission issues, please make sure that your username has write access on the database.

$ chown username /path/to/pgsql
Database authentication#

The database authentication is configured in pg_hba.conf. It can be either set to ident sameuser or md5. If set to md5, make sure to use an existing user of your database. If set to ident sameuser, make sure that your client’s operating system user name has a matching user in the database. If not, please do as follow to create a user:

$ su
$ su - postgres
$ createuser -s -P <dbuser>

The option -P (for password prompt), will encrypt the password with the method set in the configuration file pg_hba.conf. If you do not use this option -P, then the default value will be null and you will need to set it with:

$ su postgres -c "echo ALTER USER <dbuser> WITH PASSWORD '<dbpassword>' | psql"

The above login/password will be requested when you will create an instance with cubicweb-ctl create to initialize the database of your instance.

Database creation#

If you create the database by hand (instead of using the cubicweb-ctl db-create tool), you may want to make sure that the local settings are properly set. For example, if you need to handle french accents properly for indexing and sorting, you may need to create the database with something like:

$ createdb --encoding=UTF-8 --locale=fr_FR.UTF-8 -t template0 -O <owner> <dbname>

Notice that the cubicweb-ctl db-create does database initialization that may requires a postgres superuser. That’s why a login/password is explicitly asked at this step, so you can use there a superuser without using this user when running the instance. Things that require special privileges at this step:

  • database creation, require the ‘create database’ permission

Where pgadmin is a postgres superuser.

SQLite#

SQLite has the great advantage of requiring almost no configuration. Simply use ‘sqlite’ as db-driver, and set path to the dabase as db-name. Don’t specify anything for db-user and db-password, they will be ignore anyway.

Note

SQLite is great for testing and to play with cubicweb but is not suited for production environments.

Cubicweb resources configuration#

Resource mode#

Standard resource mode#

A resource mode is a predefined set of settings for various resources directories, such as cubes, instances, etc. to ease development with the framework. There are two running modes with CubicWeb:

  • system: resources are searched / created in the system directories (eg usually requiring root access):

    • instances are stored in <INSTALL_PREFIX>/etc/cubicweb.d

    • temporary files (such as pid file) in <INSTALL_PREFIX>/var/run/cubicweb

    where <INSTALL_PREFIX> is the detected installation prefix (‘/usr/local’ for instance).

  • user: resources are searched / created in the user home directory:

    • instances are stored in ~/etc/cubicweb.d

    • temporary files (such as pid file) in /tmp

Within virtual environment#

When installed within a virtualenv, CubicWeb will look for instances data as in user mode by default, that is in $HOME/etc/cubicweb.d. However the CW_INSTANCES_DIR environment variable should be preferably used.

Custom resource location#

Notice that each resource path may be explicitly set using an environment variable if the default doesn’t suit your needs. Here are the default resource directories that are affected according to mode:

  • system:

    CW_INSTANCES_DIR = <INSTALL_PREFIX>/etc/cubicweb.d/
CW_INSTANCES_DATA_DIR = <INSTALL_PREFIX>/var/lib/cubicweb/instances/
CW_RUNTIME_DIR = <INSTALL_PREFIX>/var/run/cubicweb/

  • user:

    CW_INSTANCES_DIR = ~/etc/cubicweb.d/
CW_INSTANCES_DATA_DIR = ~/etc/cubicweb.d/
CW_RUNTIME_DIR = /tmp

Cubes search path is also affected, see the Cubes section.

Setting Cubicweb Mode#

By default, the mode is set to ‘system’ for standard installation. The mode is set to ‘user’ if cubicweb is used from a mercurial repository. You can force this by setting the CW_MODE environment variable to either ‘user’ or ‘system’ so you can easily:

  • use system wide installation but user specific instances and all, without root privileges on the system (export CW_MODE=user)

  • use local checkout of cubicweb on system wide instances (requires root privileges on the system (export CW_MODE=system)

If you’ve a doubt about the mode you’re currently running, check the first line outputed by the cubicweb-ctl list command.

Development Mode (source)#

If .hg directory is found into the cubicweb package, there are specific resource rules.

<CW_SOFTWARE_ROOT> is the source checkout’s cubicweb directory:

  • cubicweb migration files are searched in <CW_SOFTWARE_ROOT>/misc/migration instead of <INSTALL_PREFIX>/share/cubicweb/migration/.

Development Mode (virtualenv)#

If a virtualenv is found to be activated (i.e. a VIRTUAL_ENV variable is found in environment), the virtualenv root is used as <INSTALL_PREFIX>. This, in particular, makes it possible to work in setuptools development mode (python setup.py develop) without any further configuration.

Environment configuration#

Python#

If you installed CubicWeb by cloning the Mercurial shell repository or from source distribution, then you will need to update the environment variable PYTHONPATH by adding the path to cubicweb:

Add the following lines to either .bashrc or .bash_profile to configure your development environment

export PYTHONPATH=/full/path/to/grshell-cubicweb

If you installed CubicWeb with packages, no configuration is required and your new cubes will be placed in /usr/share/cubicweb/cubes and your instances will be placed in /etc/cubicweb.d.

CubicWeb#

Here are all environment variables that may be used to configure CubicWeb:

CW_MODE#

Resource mode: user or system, as explained in Resource mode.

CW_INSTANCES_DIR#

Directory where cubicweb instances will be found.

CW_INSTANCES_DATA_DIR#

Directory where cubicweb instances data will be written (backup file…)

CW_RUNTIME_DIR#

Directory where pid files will be written

Deploy a CubicWeb application#

Deployment with uwsgi#

uWSGI is often used to deploy CubicWeb applications.

Short version is install uwsgi:

apt install uwsgi

Deploy a configuration file for your application /etc/uwsgi/apps-enabled/example.ini. Don’t forget to replace example with the instance name to deploy:

[uwsgi]
master = true
http = 0.0.0.0:8080
env = CW_INSTANCE=example
module = cubicweb.pyramid:wsgi_application()
processes = 2
threads = 8
plugins = http,python3
auto-procname = true
lazy-apps = true
log-master = true
disable-logging = true

You can run it manualliy with:

uwsgi --ini /etc/uwsgi/apps-enabled/example.ini

Apache configuration#

It is possible to use apache (for example) as proxy in front of uwsgi.

For this to work you have to activate the following apache modules :

  • rewrite

  • proxy

  • http_proxy

The command on Debian based systems for that is

a2enmod rewrite http_proxy proxy
/etc/init.d/apache2 restart
Example

For an apache redirection of a site accessible via http://localhost/demo while cubicweb is actually running on port 8080::

ProxyPreserveHost On
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/demo
RewriteRule ^/demo$ /demo/
RewriteRule ^/demo/(.*) http://127.0.0.1:8080/$1 [L,P]

and we will configure the base-url in the all-in-one.conf of the instance::

base-url = http://localhost/demo

Deployment with SaltStack#

To deploy with SaltStack one can refer themselves to the cubicweb-formula.

Deployment with Docker#

To deploy in a docker container cluster, you should use our docker image. The source code is also in the forge. For a standard cube with no apt dependencies, the following dockerfile is fine:

FROM logilab/cubicweb:3.29
USER root
COPY . /src
RUN pip install -e /src
USER cubicweb
RUN docker-cubicweb-helper create-instance

To run your instance, don’t forget the port redirection and change the image name:

docker run --rm -it -p 8080:8080 example:latest

If you need to customize the variables in the files all-in-one.conf or sources, you should pass them as environnement variables. For example, the database name is read from CW_DB_NAME. The admin password is read from CW_PASSWORD. Also if the database is on the host, it has to be accessible from the container:

docker run --rm -it -p 8080:8080 --env-file ./.env -v /var/run/postgresql:/var/run/postgresql example:latest

If your instance needs a scheduler, it has to be run in a separate container from the same image:

docker run --rm -it --env-file ./.env -v /var/run/postgresql:/var/run/postgresql example:latest cubicweb-ctl scheduler instance

Don’t forget to change the image name example:latest and the instance name name.

Deployment with Kubernetes#

To deploy in a Kubernetes cluster, you can take inspiration from the deploy instructions included in the fresh cube. It includes nginx to serve static files, one container for the application and one for the scheduler and also an initContainer to automatically upgrade the database in case of new version.

cubicweb-ctl tool#

cubicweb-ctl is the swiss knife to manage CubicWeb instances. The general syntax is

cubicweb-ctl <command> [options command] <arguments commands>

To view available commands

cubicweb-ctl
cubicweb-ctl --help

Please note that the commands available depends on the CubicWeb packages and cubes that have been installed.

To view the help menu on specific command

cubicweb-ctl <command> --help

Listing available cubes and instance#

  • list, provides a list of the available configuration, cubes and instances.

Creation of a new cube#

Create your new cube cube

cubicweb-ctl newcube -d <target directory>

This will create a new cube <target directory>.

Create an instance#

You must ensure ~/etc/cubicweb.d/ exists prior to this. On windows, the ‘~’ part will probably expand to ‘Documents and Settings/user’.

To create an instance from an existing cube, execute the following command

cubicweb-ctl create <cube_name> <instance_name>

This command will create the configuration files of an instance in ~/etc/cubicweb.d/<instance_name>.

The tool cubicweb-ctl executes the command db-create and db-init when you run create so that you can complete an instance creation in a single command. But of course it is possible to issue these separate commands separately, at a later stage.

Command to create/initialize an instance database#

  • db-create, creates the system database of an instance (tables and extensions only)

  • db-init, initializes the system database of an instance (schema, groups, users, workflows…)

Run an instance#

To start an instance during development, use

cubicweb-ctl pyramid [-D] [-l <log-level>] <instance-id>

without -D, the instance will be start in the background, as a daemon.

See The ‘pyramid’ command for more details.

In production, it is recommended to run CubicWeb through a WSGI server like uWSGI or Gunicorn. See cubicweb.pyramid more details.

Commands to maintain instances#

  • upgrade, launches the existing instances migration when a new version of CubicWeb or the cubes installed is available

  • shell, opens a (Python based) migration shell for manual maintenance of the instance

  • db-dump, creates a dump of the system database

  • db-restore, restores a dump of the system database

  • db-check, checks data integrity of an instance. If the automatic correction is activated, it is recommanded to create a dump before this operation.

  • schema-sync, synchronizes the persistent schema of an instance with the instance schema. It is recommanded to create a dump before this operation.

Commands to maintain i18n catalogs#

  • i18ncubicweb, regenerates messages catalogs of the CubicWeb library

  • i18ncube, regenerates the messages catalogs of a cube

  • i18ninstance, recompiles the messages catalogs of an instance. This is automatically done while upgrading.

See also chapter Internationalization.

Other commands#

  • delete, deletes an instance (configuration files and database)

Creation of your first instance#

Instance creation#

Now that we created a cube, we can create an instance and access it via a web browser. We will use a all-in-one configuration to simplify things

cubicweb-ctl create -c all-in-one mycube myinstance

Note

Please note that we created a new cube for a demo purposes but you could have used an existing cube available in our standard library such as blog or person for example.

A series of questions will be prompted to you, the default answer is usually sufficient. You can anyway modify the configuration later on by editing configuration files. When a login/password are requested to access the database please use the credentials you created at the time you configured the database (PostgreSQL).

It is important to distinguish here the user used to access the database and the user used to login to the cubicweb instance. When an instance starts, it uses the login/password for the database to get the schema and handle low level transaction. But, when cubicweb-ctl create asks for a manager login/psswd of CubicWeb, it refers to the user you will use during the development to administrate your web instance. It will be possible, later on, to use this user to create other users for your final web instance.

Instance administration#

start / stop#

When this command is completed, the definition of your instance is located in ~/etc/cubicweb.d/myinstance/*. To launch it, you just type

cubicweb-ctl pyramid -D myinstance

The option -D specifies the debug mode : the instance is not running in server mode and does not disconnect from the terminal, which simplifies debugging in case the instance is not properly launched. You can see how it looks by visiting the URL http://localhost:8080 (the port number depends of your configuration). To login, please use the cubicweb administrator login/password you defined when you created the instance.

To shutdown the instance, Crtl-C in the terminal window is enough. If you did not use the option -D, then type

cubicweb-ctl stop myinstance

This is it! All is settled down to start developping your data model…

Note

The output of cubicweb-ctl pyramid -D myinstance can be overwhelming. It is possible to reduce the log level with the –loglevel parameter as in cubicweb-ctl pyramid -D myinstance -l info to filter out all logs under info gravity.

upgrade#

A manual upgrade step is necessary whenever a new version of CubicWeb or a cube is installed, in order to synchronise the instance’s configuration and schema with the new code. The command is:

cubicweb-ctl upgrade myinstance

A series of questions will be asked. It always starts with a proposal to make a backup of your sources (where it applies). Unless you know exactly what you are doing (i.e. typically fiddling in debug mode, but definitely NOT migrating a production instance), you should answer YES to that.

The remaining questions concern the migration steps of CubicWeb, then of the cubes that form the whole application, in reverse dependency order.

In principle, if the migration scripts have been properly written and tested, you should answer YES to all questions.

Somtimes, typically while debugging a migration script, something goes wrong and the migration fails. Unfortunately the databse may be in an incoherent state. You have two options here:

  • fix the bug, restore the database and restart the migration process from scratch (quite recommended in a production environement)

  • try to replay the migration up to the last successful commit, that is answering NO to all questions up to the step that failed, and finish by answering YES to the remaining questions.

Configure an instance#

While creating an instance, a configuration file is generated in:

$ (CW_INSTANCES_DIR) / <instance> / <configuration name>.conf

For example:

/etc/cubicweb.d/myblog/all-in-one.conf

It is a simple text file in the INI format (http://en.wikipedia.org/wiki/INI_file). In the following description, each option name is prefixed with its own section and followed by its default value if necessary, e.g. “<section>.<option> [value].”

Note

At runtime, configuration options can be overriden by environments variables which name follows the option name with - replaced by _ and a CW_ prefix. For instance CW_BASE_URL=https://www.example.com would override the base-url configuration option.

Configuring the Web server#

web.auth-model [cookie]

authentication mode, cookie or http

web.realm

realm of the instance in http authentication mode

web.http-session-time [0]

period of inactivity of an HTTP session before it closes automatically. Duration in seconds, 0 meaning no expiration (or more exactly at the closing of the browser client)

main.anonymous-user, main.anonymous-password

login and password to use to connect to the RQL server with HTTP anonymous connection. CWUser account should exist.

main.base-url

url base site to be used to generate the urls of web pages

Setting up the web client#

web.embed-allowed

regular expression matching sites which could be “embedded” in the site (controllers ‘embed’)

web.submit-url

url where the bugs encountered in the instance can be mailed to

RQL server configuration#

main.host

host name if it can not be detected correctly

main.pid-file

file where will be written the server pid

main.uid

user account to use for launching the server when it is root launched by init

main.session-time [30*60]

timeout of a RQL session

main.query-log-file

file where all requests RQL executed by the server are written

Configuring e-mail#

RQL and web server side:

email.mangle-mails [no]

indicates whether the email addresses must be displayed as is or transformed

RQL server side:

email.smtp-host [mail]

hostname hosting the SMTP server to use for outgoing mail

email.smtp-port [25]

SMTP server port to use for outgoing mail

email.smtp-username

SMTP server username for authenticated email sending

email.smtp-password

SMTP server password for authenticated email sending

email.sender-name

name to use for outgoing mail of the instance

email.sender-addr

address for outgoing mail of the instance

email.default dest-addrs

destination addresses by default, if used by the configuration of the dissemination of the model (separated by commas)

email.supervising-addrs

destination addresses of e-mails of supervision (separated by commas)

Configuring logging#

main.log-threshold

level of filtering messages (DEBUG, INFO, WARNING, ERROR)

main.log-file

file to write messages

Configuring persistent properties#

Other configuration settings are in the form of entities CWProperty in the database. It must be edited via the web interface or by RQL queries.

ui.encoding

Character encoding to use for the web

navigation.short-line-size

number of characters for “short” display

navigation.page-size

maximum number of entities to show per results page

navigation.related-limit

number of related entities to show up on primary entity view

navigation.combobox-limit

number of entities unrelated to show up on the drop-down lists of the sight on an editing entity view

Cross-Origin Resource Sharing#

CubicWeb’s support for the CORS protocol is provided by the wsgicors middleware at the Pyramid level. For now, the provided implementation only deals with access to a CubicWeb instance as a whole. Support for a finer granularity may be considered in the future.

A few parameters can be set to configure the CORS capabilities of CubicWeb, the values are passed to the wsgicors.CORS() middleware constructor, hence the wsgicors documentation can be used for more details.

access-control-allow-origin

comma-separated list of allowed origin domains or “*” for any domain

access-control-allow-methods

comma-separated list of allowed HTTP methods

access-control-allow-headers

comma-separated list of allowed HTTP custom headers (used in simple requests)

access-control-expose-headers

comma-separated list of allowed HTTP custom headers (used in preflight requests)

access-control-max-age

maximum age of cross-origin resource sharing (in seconds)

credentials is always set to true and is not configurable.

User interface for web site configuration#

_images/lax-book_03-site-config-panel_en.png

This panel allows you to configure the appearance of your instance site. Six menus are available and we will go through each of them to explain how to use them.

UI#

This menu provides you a way to customize the user interface settings such as date format or encoding in the produced html. Follows the detailled list of available options :

  • ui.date-format : how to format date in the ui (“man strftime” for format description)

  • ui.datetime-format : how to format date and time in the ui (“man strftime” for format description)

  • ui.default-text-format : default text format for rich text fields.

  • ui.encoding : user interface encoding

  • ui.fckeditor :should html fields being edited using fckeditor (a HTML WYSIWYG editor). You should also select text/html as default text format to actually get fckeditor.

  • ui.float-format : how to format float numbers in the ui

  • ui.language : language of the user interface

  • ui.main-template : id of main template used to render pages

  • ui.site-title : site title, which is displayed right next to the logo in the header

  • ui.time-format : how to format time in the ui (“man strftime” for format description)

Actions#

This menu provides a way to configure the context in which you expect the actions to be displayed to the user and if you want the action to be visible or not. You must have notice that when you view a list of entities, an action box is available on the left column which display some actions as well as a drop-down menu for more actions.

The context available are :

  • mainactions : actions listed in the left box

  • moreactions : actions listed in the more menu of the left box

  • addrelated : add actions listed in the left box

  • useractions : actions listed in the first section of drop-down menu accessible from the right corner user login link

  • siteactions : actions listed in the second section of drop-down menu accessible from the right corner user login link

  • hidden : select this to hide the specific action

Boxes#

The instance has already a pre-defined set of boxes you can use right away. This configuration section allows you to place those boxes where you want in the instance interface to customize it.

The available boxes are :

  • actions box : box listing the applicable actions on the displayed data

  • boxes_blog_archives_box : box listing the blog archives

  • possible views box : box listing the possible views for the displayed data

  • rss box : RSS icon to get displayed data as a RSS thread

  • search box : search box

  • startup views box : box listing the configuration options available for the instance site, such as Preferences and Site Configuration

Components#

[WRITE ME]

Contextual components#

[WRITE ME]

Multiple sources of data#

Data sources include SQL, LDAP, RQL, mercurial and subversion.

LDAP integration#

Overview#

Using LDAP as a source for user credentials and information is quite easy. The most difficult part lies in building an LDAP schema or using an existing one.

At cube creation time, one is asked if more sources are wanted. LDAP is one possible option at this time. Of course, it is always possible to set it up later using the CWSource entity type, which we discuss there.

It is possible to add as many LDAP sources as wanted, which translates in as many CWSource entities as needed.

The general principle of the LDAP source is, given a proper configuration, to create local users matching the users available in the directory and deriving local user attributes from directory users attributes. Then a periodic task ensures local user information synchronization with the directory.

Users handled by such a source should not be edited directly from within the application instance itself. Rather, updates should happen at the LDAP server level.

Credential checks are _always_ done against the LDAP server.

Note

There are currently two ldap source types: the older ldapuser and the newer ldapfeed. The older will be deprecated anytime soon, as the newer has now gained all the features of the old and does not suffer from some of its illnesses.

The ldapfeed creates real CWUser entities, and then activate/deactivate them depending on their presence/absence in the corresponding LDAP source. Their attribute and state (activated/deactivated) are hence managed by the source mechanism; they should not be altered by other means (as such alterations may be overridden in some subsequent source synchronisation).

Configuration of an LDAPfeed source#

Additional sources are created at cube creation time or later through the user interface.

Configure an ldapfeed source from the user interface under Manage then data sources:

  • At this point type has been set to ldapfeed.

  • The parser attribute shall be set to ldapfeed.

  • The url attribute shall be set to an URL such as ldap://ldapserver.domain/.

  • The configuration attribute contains many options. They are described in detail in the next paragraph.

Options of an LDAPfeed source#

Let us enumerate the options by categories (LDAP server connection, LDAP schema mapping information).

LDAP server connection options:

  • auth-mode, (choices are simple, cram_md5, digest_md5, gssapi, support for the later being partial as of now)

  • auth-realm, realm to use when using gssapi/kerberos authentication

  • data-cnx-dn, user dn to use to open data connection to the ldap (eg used to respond to rql queries)

  • data-cnx-password, password to use to open data connection to the ldap (eg used to respond to rql queries)

  • start-tls, starting TLS before bind (valid values: “true”, “false”)

If the LDAP server accepts anonymous binds, then it is possible to leave data-cnx-dn and data-cnx-password empty. This is, however, quite unlikely in practice. Beware that the LDAP server might hide attributes such as “userPassword” while the rest of the attributes remain visible through an anonymous binding.

LDAP schema mapping options:

  • user-base-dn, base DN to lookup for users

  • user-scope, user search scope (valid values: “BASE”, “ONELEVEL”, “SUBTREE”)

  • user-classes, classes of user (with Active Directory, you want to say “user” here)

  • user-filter, additional filters to be set in the ldap query to find valid users

  • user-login-attr, attribute used as login on authentication (with Active Directory, you want to use “sAMAccountName” here)

  • user-default-group, name of a group in which ldap users will be by default. You can set multiple groups by separating them by a comma

  • user-attrs-map, map from ldap user attributes to cubicweb attributes (with Active Directory, you want to use sAMAccountName:login,mail:email,givenName:firstname,sn:surname)

Other notes#

  • Cubicweb is able to start if ldap cannot be reached… If some source ldap server cannot be used while an instance is running, the corresponding users won’t be authenticated but their status will not change (e.g. they will not be deactivated)

  • The user-base-dn is a key that helps cubicweb map CWUsers to LDAP users: beware updating it

  • When a user is removed from an LDAP source, it is deactivated in the CubicWeb instance; when a deactivated user comes back in the LDAP source, it (automatically) is activated again

  • You can use the CWSourceHostConfig to have variants for a source configuration according to the host the instance is running on. To do so go on the source’s view from the sources management view.

RQL logs#

You can configure the CubicWeb instance to keep a log of the queries executed against your database. To do so, edit the configuration file of your instance .../etc/cubicweb.d/myapp/all-in-one.conf and uncomment the variable query-log-file:

# web instance query log file
query-log-file=/tmp/rql-myapp.log

Web Frontend Development#

In this chapter, we will describe the core APIs for web development in the CubicWeb framework.

Publisher#

What happens when an HTTP request is issued ?

The story begins with the CubicWebPublisher.main_publish method. We do not get upper in the bootstrap process because it is dependant on the used HTTP library.

What main_publish does:

  • get a controller id and a result set from the path (this is actually delegated to the urlpublisher component)

  • the controller is then selected (if not, this is considered an authorization failure and signaled as such) and called

  • then either a proper result is returned, in which case the request/connection object issues a commit and returns the result

  • or error handling must happen:

    • ValidationErrors pop up there and may lead to a redirect to a previously arranged url or standard error handling applies

    • an HTTP 500 error (Internal Server Error) is issued

Now, let’s turn to the controller. There are many of them in cubicweb.web.views.basecontrollers. We can just follow the default view controller that is selected on a view path. See the Controllers chapter for more information on controllers.

The View controller’s entry point is the publish method. It does the following:

  • compute the main view to be applied, using either the given result set or building one from a user provided rql string (rql and vid can be forced from the url GET parameters), that is:

    • compute the vid using the result set and the schema (see cubicweb.web.views.vid_from_rset)

    • handle all error cases that could happen in this phase

  • do some cache management chores

  • select a main template (typically TheMainTemplate, see chapter Templates)

  • call it with the result set and the computed view.

What happens next actually depends on the template and the view, but in general this is the rendering phase.

CubicWebPublisher API#

Controllers#

Overview#

Controllers are responsible for taking action upon user requests (loosely following the terminology of the MVC meta pattern).

The following controllers are provided out-of-the box in CubicWeb. We list them by category. They are all defined in (cubicweb.web.views.basecontrollers).

Browsing:

  • the View controller is associated with most browsing actions within a CubicWeb application: it always instantiates a TheMainTemplate and lets the ResultSet/Views dispatch system build up the whole content; it handles ObjectNotFound and NoSelectableObject errors that may bubble up to its entry point, in an end-user-friendly way (but other programming errors will slip through)

  • the JSonpController is a wrapper around the ViewController that provides jsonp services. Padding can be specified with the callback request parameter. Only jsonexport / ejsonexport views can be used. If another vid is specified, it will be ignored and replaced by jsonexport. Request is anonymized to avoid returning sensitive data and reduce the risks of CSRF attacks;

  • the Login/Logout controllers make effective user login or logout requests

Edition:

  • the Edit controller (see The edit controller) handles CRUD operations in response to a form being submitted; it works in close association with the Forms, to which it delegates some of the work

  • the Form validator controller provides form validation from Ajax context, using the Edit controller, to implement the classic form handling loop (user edits, hits submit/apply, validation occurs server-side by way of the Form validator controller, and the UI is decorated with failure information, either global or per-field , until it is valid)

Other:

  • the SendMail controller (web/views/basecontrollers.py) is reponsible for outgoing email notifications

  • the MailBugReport controller (web/views/basecontrollers.py) allows to quickly have a reportbug feature in one’s application

  • the cubicweb.web.views.ajaxcontroller.AjaxController (cubicweb.web.views.ajaxcontroller) provides services for Ajax calls, typically using JSON as a serialization format for input, and sometimes using either JSON or XML for output. See Ajax chapter for more information.

Registration#

All controllers (should) live in the ‘controllers’ namespace within the global registry.

Concrete controllers#

Most API details should be resolved by source code inspection, as the various controllers have differing goals. See for instance the The edit controller chapter.

cubicweb.web.controller contains the top-level abstract Controller class and its unimplemented entry point publish(rset=None) method.

A handful of helpers are also provided there:

  • process_rql builds a result set from an rql query typically issued from the browser (and available through _cw.form[‘rql’])

  • validate_cache will force cache validation handling with respect to the HTTP Cache directives (that were typically originally issued from a previous server -> client response); concrete Controller implementations dealing with HTTP (thus, for instance, not the SendMail controller) may very well call this in their publication process.

The Request class (cubicweb.web.request)#

Overview#

A request instance is created when an HTTP request is sent to the web server. It contains informations such as form parameters, authenticated user, etc. It is a very prevalent object and is used throughout all of the framework and applications, as you’ll access to almost every resources through it.

A request represents a user query, either through HTTP or not (we also talk about RQL queries on the server side for example).

Here is a non-exhaustive list of attributes and methods available on request objects (grouped by category):

  • Browser control:

    • ie_browser: tells if the browser belong to the Internet Explorer family

  • User and identification:

    • user, instance of cubicweb.entities.authobjs.CWUser corresponding to the authenticated user

  • Session data handling

    • session.data is the dictionary of the session data; it can be manipulated like an ordinary Python dictionary

  • Edition (utilities for edition control):

    • cancel_edition: resets error url and cleans up pending operations

    • create_entity: utility to create an entity (from an etype, attributes and relation values)

    • datadir_url: returns the url to the merged external resources (CubicWeb’s web/data directory plus all data directories of used cubes)

    • edited_eids: returns the list of eids of entities that are edited under the current http request

    • eid_rset(eid): utility which returns a result set from an eid

    • entity_from_eid(eid): returns an entity instance from the given eid

    • encoding: returns the encoding of the current HTTP request

    • ensure_ro_rql(rql): ensure some rql query is a data request

    • etype_rset

    • form, dictionary containing the values of a web form

    • encoding, character encoding to use in the response

  • HTTP

    • authmode: returns a string describing the authentication mode (http, cookie, …)

    • lang: returns the user agents/browser’s language as carried by the http request

    • demote_to_html(): in the context of an XHTML compliant browser, this will force emission of the response as an HTML document (using the http content negociation)

  • Cookies handling

  • get_cookie(), returns a dictionary containing the value of the header HTTP ‘Cookie’

  • set_cookie(cookie, key, maxage=300), adds a header HTTP Set-Cookie, with a minimal 5 minutes length of duration by default (maxage = None returns a session cookie which will expire when the user closes the browser window)

  • remove_cookie(cookie, key), forces a value to expire

  • URL handling

    • build_url(__vid, *args, **kwargs): return an absolute URL using params dictionary key/values as URL parameters. Values are automatically URL quoted, and the publishing method to use may be specified or will be guessed.

    • build_url_params(**kwargs): returns a properly prepared (quoted, separators, …) string from the given parameters

    • url(), returns the full URL of the HTTP request

    • base_url(), returns the root URL of the web application

    • relative_path(), returns the relative path of the request

  • Web resource (.css, .js files, etc.) handling:

    • add_css(cssfiles): adds the given list of css resources to the current html headers

    • add_js(jsfiles): adds the given list of javascript resources to the current html headers

    • add_onload(jscode): inject the given jscode fragment (a unicode string) into the current html headers, wrapped inside a document.ready(…) or another ajax-friendly one-time trigger event

    • add_header(header, values): adds the header/value pair to the current html headers

    • status_out: control the HTTP status of the response

  • And more…

    • set_content_type(content_type, filename=None), adds the header HTTP ‘Content-Type’

    • get_header(header), returns the value associated to an arbitrary header of the HTTP request

    • set_header(header, value), adds an arbitrary header in the response

    • execute(*args, **kwargs), executes an RQL query and return the result set

    • property_value(key), properties management (CWProperty)

    • dictionary data to store data to share informations between components while a request is executed

Please note that this class is abstract and that a concrete implementation will be provided by the frontend web used. For the views or others that are executed on the server side, most of the interface of Request is defined in the session associated to the client.

API#

The elements we gave in overview for above are built in three layers, from cubicweb.req.RequestSessionBase, cubicweb.repoapi.Connection and cubicweb.web.ConnectionCubicWebRequestBase.

class cubicweb.req.RequestSessionBase(*args, **kwargs)#
class cubicweb.repoapi.Connection(repo, user)[source]#

Repository Connection

Holds all connection related data

Database connection resources:

hooks_in_progress, boolean flag telling if the executing query is coming from a repoapi connection or is a query from within the repository (e.g. started by hooks)

cnxset, the connections set to use to execute queries on sources. If the transaction is read only, the connection set may be freed between actual queries. This allows multiple connections with a reasonably low connection set pool size.

mode, string telling the connections set handling mode, may be one of ‘read’ (connections set may be freed), ‘write’ (some write was done in the connections set, it can’t be freed before end of the transaction), ‘transaction’ (we want to keep the connections set during all the transaction, with or without writing)

Shared data:

data is a dictionary bound to the underlying session, who will be present for the life time of the session. This may be useful for web clients that rely on the server for managing bits of session-scoped data.

transaction_data is a dictionary cleared at the end of the transaction. Hooks and operations may put arbitrary data in there.

Internal state:

pending_operations, ordered list of operations to be processed on commit/rollback

commit_state, describing the transaction commit state, may be one of None (not yet committing), ‘precommit’ (calling precommit event on operations), ‘postcommit’ (calling postcommit event on operations), ‘uncommitable’ (some ValidationError or Unauthorized error has been raised during the transaction and so it must be rolled back).

Hooks controls:

deny_all_hooks_but(*categories)[source]#

Context manager to disable all hooks but those in the given categories.

allow_all_hooks_but(*categories)[source]#

Context manager to enable all hooks but those in the given categories.

Security level Management:

read_security and write_security, boolean flags telling if read/write security is currently activated.

add_operation(operation, index=None)[source]#

add an operation to be executed at the end of the transaction

add_relation(fromeid, rtype, toeid)[source]#

provide direct access to the repository method to add a relation.

This is equivalent to the following rql query:

SET X rtype Y WHERE X eid fromeid, T eid toeid

without read security check but also all the burden of rql execution. You may use this in hooks when you know both eids of the relation you want to add.

add_relations(relations)[source]#

set many relation using a shortcut similar to the one in add_relation

relations is a list of 2-uples, the first element of each 2-uple is the rtype, and the second is a list of (fromeid, toeid) tuples

added_in_transaction(eid)[source]#

return True if the entity of the given eid is being created in the current transaction

allow_all_hooks_but(*categories)[source]#

Context manager to enable all hooks but those in the given categories.

cached_entities()[source]#

return the whole entity cache

clear()[source]#

reset internal data

commit()[source]#

commit the current session’s transaction

commit_state#

(None, ‘precommit’, ‘postcommit’, ‘uncommitable’)

critical(msg, *args, **kwargs)#

Log ‘msg % args’ with severity ‘CRITICAL’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.critical(“Houston, we have a %s”, “major disaster”, exc_info=1)

debug(msg, *args, **kwargs)#

Log ‘msg % args’ with severity ‘DEBUG’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.debug(“Houston, we have a %s”, “thorny problem”, exc_info=1)

delete_relation(fromeid, rtype, toeid)[source]#

provide direct access to the repository method to delete a relation.

This is equivalent to the following rql query:

DELETE X rtype Y WHERE X eid fromeid, T eid toeid

without read security check but also all the burden of rql execution. You may use this in hooks when you know both eids of the relation you want to delete.

deleted_in_transaction(eid)[source]#

return True if the entity of the given eid is being deleted in the current transaction

deny_all_hooks_but(*categories)[source]#

Context manager to disable all hooks but those in the given categories.

drop_entity_cache()[source]#

Drop the whole entity cache.

entity_cache(eid)[source]#

get cache entity for eid

entity_type(eid)[source]#

Return entity type for the entity with id eid.

error(msg, *args, **kwargs)#

Log ‘msg % args’ with severity ‘ERROR’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.error(“Houston, we have a %s”, “major problem”, exc_info=1)

exception(msg, *args, exc_info=True, **kwargs)#

Convenience method for logging an ERROR with exception information.

execute(rql, kwargs=None, build_descr=True)[source]#

db-api like method directly linked to the querier execute method.

See cubicweb.dbapi.Cursor.execute() documentation.

get_option_value(option)[source]#

Return the value for option in the configuration.

get_schema()[source]#

Return the schema currently used by the repository.

info(msg, *args, **kwargs)#

Log ‘msg % args’ with severity ‘INFO’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.info(“Houston, we have a %s”, “interesting problem”, exc_info=1)

is_hook_activated(hook)[source]#

return a boolean telling if the given hook class is currently activated or not

is_hook_category_activated(category)[source]#

return a boolean telling if the given category is currently activated or not

pending_operations#

ordered list of operations to be processed on commit/rollback

repo#

server.Repository object

rollback()[source]#

rollback the current transaction

running_hooks_ops()[source]#

this context manager should be called whenever hooks or operations are about to be run (but after hook selection)

It will help the undo logic record pertinent metadata or some hooks to run (or not) depending on who/what issued the query.

set_entity_cache(entity)[source]#

Add entity to the connection entity cache

system_sql(sql, args=None, rollback_on_failure=True, rql_query_tracing_token=None)[source]#

return a sql cursor on the system database

transaction_actions(txuuid, public=True)[source]#

Return an ordered list of actions effectued during that transaction.

If public is true, return only ‘public’ actions, i.e. not ones triggered under the cover by hooks, else return all actions.

raise NoSuchTransaction if the transaction is not found or if the user is not allowed (eg not in managers group).

transaction_data#

dict containing arbitrary data cleared at the end of the transaction

transaction_info(txuuid)[source]#

Return transaction object for the given uid.

raise NoSuchTransaction if not found or if session’s user is not allowed (eg not in managers group and the transaction doesn’t belong to him).

undo_transaction(txuuid)[source]#

Undo the given transaction. Return potential restoration errors.

raise NoSuchTransaction if not found or if user is not allowed (eg not in managers group).

undoable_transactions(ueid=None, **actionfilters)[source]#

Return a list of undoable transaction objects by the connection’s user, ordered by descendant transaction time.

Managers may filter according to user (eid) who has done the transaction using the ueid argument. Others will only see their own transactions.

Additional filtering capabilities is provided by using the following named arguments:

  • etype to get only transactions creating/updating/deleting entities of the given type

  • eid to get only transactions applied to entity of the given eid

  • action to get only transactions doing the given action (action in ‘C’, ‘U’, ‘D’, ‘A’, ‘R’). If etype, action can only be ‘C’, ‘U’ or ‘D’.

  • public: when additional filtering is provided, they are by default only searched in ‘public’ actions, unless a public argument is given and set to false.

warning(msg, *args, **kwargs)#

Log ‘msg % args’ with severity ‘WARNING’.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

logger.warning(“Houston, we have a %s”, “bit of a problem”, exc_info=1)

The View system#

This chapter aims to describe the concept of a view used all along the development of a web application and how it has been implemented in CubicWeb.

Principles#

We’ll start with a description of the interface providing a basic understanding of the available classes and methods, then detail the view selection principle.

A View is an object responsible for the rendering of data from the model into an end-user consummable form. They typically churn out an XHTML stream, but there are views concerned with email other non-html outputs.

Discovering possible views#

It is possible to configure the web user interface to have a left box showing all the views than can be applied to the current result set.

To enable this, click on your login at the top right corner. Chose “user preferences”, then “boxes”, then “possible views box” and check “visible = yes” before validating your changes.

The views listed there we either not selected because of a lower score, or they were deliberately excluded by the main template logic.

Basic class for views#

Class View#

The basic interface for views is as follows (remember that the result set has a tabular structure with rows and columns, hence cells):

  • render(**context), render the view by calling call or cell_call depending on the context

  • call(**kwargs), call the view for a complete result set or null (the default implementation calls cell_call() on each cell of the result set)

  • cell_call(row, col, **kwargs), call the view for a given cell of a result set (row and col being integers used to access the cell)

  • url(), returns the URL enabling us to get the view with the current result set

  • wview(__vid, rset, __fallback_vid=None, **kwargs), call the view of identifier __vid on the given result set. It is possible to give a fallback view identifier that will be used if the requested view is not applicable to the result set.

  • html_headers(), returns a list of HTML headers to be set by the main template

  • page_title(), returns the title to use in the HTML header title

Other basic view classes#

Here are some of the subclasses of View defined in cubicweb.view that are more concrete as they relate to data rendering within the application:

Examples of views class#

  • Using templatable, content_type and HTTP cache configuration

class RSSView(XMLView):
    __regid__ = 'rss'
    title = _('rss')
    templatable = False
    content_type = 'text/xml'
    http_cache_manager = MaxAgeHTTPCacheManager
    cache_max_age = 60*60*2 # stay in http cache for 2 hours by default

  • Using a custom selector

class SearchForAssociationView(EntityView):
    """view called by the edition view when the user asks
    to search for something to link to the edited eid
    """
    __regid__ = 'search-associate'
    title = _('search for association')
    __select__ = one_line_rset() & match_search_state('linksearch') & is_instance('Any')

XML views, binaries views…#

For views generating other formats than HTML (an image generated dynamically for example), and which can not simply be included in the HTML page generated by the main template (see above), you have to:

  • set the attribute templatable of the class to False

  • set, through the attribute content_type of the class, the MIME type generated by the view to application/octet-stream or any relevant and more specialised mime type

For views dedicated to binary content creation (like dynamically generated images), we have to set the attribute binary of the class to True (which implies that templatable == False, so that the attribute w of the view could be replaced by a binary flow instead of unicode).

Templates#

Templates are the entry point for the CubicWeb view system. As seen in Discovering possible views, there are two kinds of views: the templatable and non-templatable.

Non-templatable views#

Non-templatable views are standalone. They are responsible for all the details such as setting a proper content type (or mime type), the proper document headers, namespaces, etc. Examples are pure xml views such as RSS or Semantic Web views (SIOC, DOAP, FOAF, Linked Data, etc.), and views which generate binary files (pdf, excel files, etc.)

To notice that a view is not templatable, you just have to set the view’s class attribute templatable to False. In this case, it should set the content_type class attribute to the correct MIME type. By default, it is text/xhtml. Additionally, if your view generate a binary file, you have to set the view’s class attribute binary to True too.

Templatable views#

Templatable views are not concerned with such pesky details. They leave it to the template. Conversely, the template’s main job is to:

  • set up the proper document header and content type

  • define the general layout of a document

  • invoke adequate views in the various sections of the document

Look at cubicweb.web.views.basetemplates and you will find the base templates used to generate (X)HTML for your application. The most important template there is TheMainTemplate.

TheMainTemplate#
Layout and sections#

A page is composed as indicated on the schema below :

_images/main_template.png

The sections dispatches specific views:

  • header: the rendering of the header is delegated to the htmlheader view, whose default implementation can be found in basetemplates.py and which does the following things:

    • inject the favicon if there is one

    • inject the global style sheets and javascript resources

    • call and display a link to an rss component if there is one available

    it also sets up the page title, and fills the actual header section with top-level components, using the header view, which:

    • tries to display a logo, the name of the application and the breadcrumbs

    • provides a login status area

    • provides a login box (hiden by default)

  • left column: this is filled with all selectable boxes matching the left context (there is also a right column but nowadays it is seldom used due to bad usability)

  • contentcol: this is the central column; it is filled with:

    • the rqlinput view (hidden by default)

    • the applmessages component

    • the contentheader view which in turns dispatches all available content navigation components having the navtop context (this is used to navigate through entities implementing the IPrevNext interface)

    • the view that was given as input to the template’s call method, also dealing with pagination concerns

    • the contentfooter

  • footer: adds all footer actions

Note

How and why a view object is given to the main template is explained in the Publisher chapter.

Configure the main template#

You can overload some methods of the TheMainTemplate, in order to fulfil your needs. There are also some attributes and methods which can be defined on a view to modify the base template behaviour:

  • paginable: if the result set is bigger than a configurable size, your result page will be paginated by default. You can set this attribute to False to avoid this.

  • binary: boolean flag telling if the view generates some text or a binary stream. Default to False. When view generates text argument given to self.w must be a unicode string, encoded string otherwise.

  • content_type, view’s content type, default to ‘text/xhtml’

  • templatable, boolean flag telling if the view’s content should be returned directly (when False) or included in the main template layout (including header, boxes and so on).

  • page_title(), method that should return a title that will be set as page title in the html headers.

  • html_headers(), method that should return a list of HTML headers to be included the html headers.

You can also modify certain aspects of the main template of a page when building a url or setting these parameters in the req.form:

  • __notemplate, if present (whatever the value assigned), only the content view is returned

  • __force_display, if present and its value is not null, no pagination whatever the number of entities to display (e.g. similar effect as view’s paginable attribute described above.

  • __method, if the result set to render contains only one entity and this parameter is set, it refers to a method to call on the entity by passing it the dictionary of the forms parameters, before going the classic way (through step 1 and 2 described juste above)

  • vtitle, a title to be set as <h1> of the content

Other templates#

There are also the following other standard templates:

  • cubicweb.web.views.basetemplates.LogInTemplate

  • cubicweb.web.views.basetemplates.LogOutTemplate

  • cubicweb.web.views.basetemplates.ErrorTemplate specializes TheMainTemplate to do proper end-user output if an error occurs during the computation of TheMainTemplate (it is a fallback view).

The Primary View#

By default, CubicWeb provides a view that fits every available entity type. This is the first view you might be interested in modifying. It is also one of the richest and most complex.

It is automatically selected on a one line result set containing an entity.

It lives in the cubicweb.web.views.primary module.

The primary view is supposed to render a maximum of informations about the entity.

Layout#

The primary view has the following layout.

_images/primaryview_template.png

Primary view configuration#

If you want to customize the primary view of an entity, overriding the primary view class may not be necessary. For simple adjustments (attributes or relations display locations and styles), a much simpler way is to use uicfg.

Attributes/relations display location#

In the primary view, there are three sections where attributes and relations can be displayed (represented in pink in the image above):

  • ‘attributes’

  • ‘relations’

  • ‘sideboxes’

Attributes can only be displayed in the attributes section (default

behavior). They can also be hidden. By default, attributes of type Password and Bytes are hidden.

For instance, to hide the title attribute of the Blog entity:

from cubicweb.web.views import uicfg
uicfg.primaryview_section.tag_attribute(('Blog', 'title'), 'hidden')

Relations can be either displayed in one of the three sections or hidden.

For relations, there are two methods:

  • tag_object_of for modifying the primary view of the object

  • tag_subject_of for modifying the primary view of the subject

These two methods take two arguments:

  • a triplet (subject, relation_name, object), where subject or object can be replaced with '*'

  • the section name or hidden

pv_section = uicfg.primaryview_section
# hide every relation `entry_of` in the `Blog` primary view
pv_section.tag_object_of(('*', 'entry_of', 'Blog'), 'hidden')

# display `entry_of` relations in the `relations`
# section in the `BlogEntry` primary view
pv_section.tag_subject_of(('BlogEntry', 'entry_of', '*'), 'relations')
Display content#

You can use primaryview_display_ctrl to customize the display of attributes or relations. Values of primaryview_display_ctrl are dictionaries.

Common keys for attributes and relations are:

  • vid: specifies the regid of the view for displaying the attribute or the relation.

    If vid is not specified, the default value depends on the section:

    • attributes section: ‘reledit’ view

    • relations section: ‘autolimited’ view

    • sideboxes section: ‘sidebox’ view

  • order: int used to control order within a section. When not specified, automatically set according to order in which tags are added.

  • label: label for the relations section or side box

  • showlabel: boolean telling whether the label is displayed

# let us remind the schema of a blog entry
class BlogEntry(EntityType):
    title = String(required=True, fulltextindexed=True, maxsize=256)
    publish_date = Date(default='TODAY')
    content = String(required=True, fulltextindexed=True)
    entry_of = SubjectRelation('Blog', cardinality='?*')

# now, we want to show attributes
# with an order different from that in the schema definition
view_ctrl = uicfg.primaryview_display_ctrl
for index, attr in enumerate('title', 'content', 'publish_date'):
    view_ctrl.tag_attribute(('BlogEntry', attr), {'order': index})

By default, relations displayed in the ‘relations’ section are being displayed by the ‘autolimited’ view. This view will use comma separated values, or list view and/or limit your rset if there is too much items in it (and generate the “view all” link in this case).

You can control this view by setting the following values in the primaryview_display_ctrl relation tag:

  • limit, maximum number of entities to display. The value of the ‘navigation.related-limit’ cwproperty is used by default (which is 8 by default). If None, no limit.

  • use_list_limit, number of entities until which they should be display as a list (eg using the ‘list’ view). Below that limit, the ‘csv’ view is used. If None, display using ‘csv’ anyway.

  • subvid, the subview identifier (eg view that should be used of each item in the list)

Notice you can also use the filter key to set up a callback taking the related result set as argument and returning it filtered, to do some arbitrary filtering that can’t be done using rql for instance.

pv_section = uicfg.primaryview_section
# in `CWUser` primary view, display `created_by`
# relations in relations section
pv_section.tag_object_of(('*', 'created_by', 'CWUser'), 'relations')

# display this relation as a list, sets the label,
# limit the number of results and filters on comments
def filter_comment(rset):
    return rset.filtered_rset(lambda x: x.e_schema == 'Comment')
pv_ctrl = uicfg.primaryview_display_ctrl
pv_ctrl.tag_object_of(('*', 'created_by', 'CWUser'),
                      {'vid': 'list', 'label': _('latest comment(s):'),
                       'limit': True,
                       'filter': filter_comment})

Warning

with the primaryview_display_ctrl rtag, the subject or the object of the relation is ignored for respectively tag_object_of or tag_subject_of. To avoid warnings during execution, they should be set to '*'.

Example of customization and creation#

We’ll show you now an example of a primary view and how to customize it.

If you want to change the way a BlogEntry is displayed, just override the method cell_call() of the view primary in BlogDemo/views.py.

from cubicweb.predicates import is_instance
from cubicweb.web.views.primary import Primaryview

class BlogEntryPrimaryView(PrimaryView):
    __select__ = PrimaryView.__select__ & is_instance('BlogEntry')

    def render_entity_attributes(self, entity):
        self.w(u'<p>published on %s</p>' %
               entity.publish_date.strftime('%Y-%m-%d'))
        super(BlogEntryPrimaryView, self).render_entity_attributes(entity)

The above source code defines a new primary view for BlogEntry. The __reid__ class attribute is not repeated there since it is inherited through the primary.PrimaryView class.

The selector for this view chains the selector of the inherited class with its own specific criterion.

The view method self.w() is used to output data. Here lines 08-09 output HTML for the publication date of the entry.

blog entries now look much nicer

Let us now improve the primary view of a blog

from logilab.mtconverter import xml_escape
from cubicweb.predicates import is_instance, one_line_rset
from cubicweb.web.views.primary import Primaryview

class BlogPrimaryView(PrimaryView):
    __regid__ = 'primary'
    __select__ = PrimaryView.__select__ & is_instance('Blog')
    rql = 'Any BE ORDERBY D DESC WHERE BE entry_of B, BE publish_date D, B eid %(b)s'

    def render_entity_relations(self, entity):
        rset = self._cw.execute(self.rql, {'b' : entity.eid})
        for entry in rset.entities():
            self.w(u'<p>%s</p>' % entry.view('inblogcontext'))

class BlogEntryInBlogView(EntityView):
    __regid__ = 'inblogcontext'
    __select__ = is_instance('BlogEntry')

    def cell_call(self, row, col):
        entity = self.cw_rset.get_entity(row, col)
        self.w(u'<a href="%s" title="%s">%s</a>' %
               entity.absolute_url(),
               xml_escape(entity.content[:50]),
               xml_escape(entity.description))

This happens in two places. First we override the render_entity_relations method of a Blog’s primary view. Here we want to display our blog entries in a custom way.

At line 10, a simple request is made to build a result set with all the entities linked to the current Blog entity by the relationship entry_of. The part of the framework handling the request knows about the schema and infers that such entities have to be of the BlogEntry kind and retrieves them (in the prescribed publish_date order).

The request returns a selection of data called a result set. Result set objects have an .entities() method returning a generator on requested entities (going transparently through the ORM layer).

At line 13 the view ‘inblogcontext’ is applied to each blog entry to output HTML. (Note that the ‘inblogcontext’ view is not defined whatsoever in CubicWeb. You are absolutely free to define whole view families.) We juste arrange to wrap each blogentry output in a ‘p’ html element.

Next, we define the ‘inblogcontext’ view. This is NOT a primary view, with its well-defined sections (title, metadata, attribtues, relations/boxes). All a basic view has to define is cell_call.

Since views are applied to result sets which can be tables of data, we have to recover the entity from its (row,col)-coordinates (line 20). Then we can spit some HTML.

Warning

Be careful: all strings manipulated in CubicWeb are actually unicode strings. While web browsers are usually tolerant to incoherent encodings they are being served, we should not abuse it. Hence we have to properly escape our data. The xml_escape() function has to be used to safely fill (X)HTML elements from Python unicode strings.

Assuming we added entries to the blog titled MyLife, displaying it now allows to read its description and all its entries.

a blog and all its entries

The “Click and Edit” (also reledit) View#

The principal way to update data through the Web UI is through the modify action on entities, which brings a full form. This is described in the HTML form construction chapter.

There is however another way to perform piecewise edition of entities and relations, using a specific reledit (for relation edition) view from the cubicweb.web.views.reledit module.

This is typically applied from the default Primary View (see The Primary View) on the attributes and relation section. It makes small editions more convenient.

Of course, this can be used customely in any other view. Here come some explanation about its capabilities and instructions on the way to use it.

Using reledit#

Let’s start again with a simple example:

class Company(EntityType):
     name = String(required=True, unique=True)
     boss = SubjectRelation('Person', cardinality='1*')
     status = SubjectRelation('File', cardinality='?*', composite='subject')

In some view code we might want to show these attributes/relations and allow the user to edit each of them in turn without having to leave the current page. We would write code as below:

company.view('reledit', rtype='name', default_value='<name>') # editable name attribute
company.view('reledit', rtype='boss') # editable boss relation
company.view('reledit', rtype='status') # editable attribute-like relation

If one wanted to edit the company from a boss’s point of view, one would have to indicate the proper relation’s role. By default the role is subject.

person.view('reledit', rtype='boss', role='object')

Each of these will provide with a different editing widget. The name attribute will obviously get a text input field. The boss relation will be edited through a selection box, allowing to pick another Person as boss. The status relation, given that it defines Company as a composite entity with one file inside, will provide additional actions

  • to add a File when there is one

  • to delete the File (if the cardinality allows it)

Moreover, editing the relation or using the add action leads to an embedded edition/creation form allowing edition of the target entity (which is File in our example) instead of merely allowing to choose amongst existing files.

The reledit_ctrl rtag#

The behaviour of reledited attributes/relations can be finely controlled using the reledit_ctrl rtag, defined in cubicweb.web.views.uicfg.

This rtag provides four control variables:

  • default_value: alternative default value

    The default value is what is shown when there is no value.

  • reload: boolean, eid (to reload to) or function taking subject

    and returning bool/eid This is useful when editing a relation (or attribute) that impacts the url or another parts of the current displayed page. Defaults to false.

  • rvid: alternative view id (as str) for relation or composite

    edition Default is ‘incontext’ or ‘csv’ depending on the cardinality. They can also be statically changed by subclassing ClickAndEditFormView and redefining _one_rvid (resp. _many_rvid).

  • edit_target: ‘rtype’ (to edit the relation) or ‘related’ (to

    edit the related entity) This controls whether to edit the relation or the target entity of the relation. Currently only one-to-one relations support target entity edition. By default, the ‘related’ option is taken whenever the relation is composite and one-to-one.

Let’s see how to use these controls.

from logilab.mtconverter import xml_escape
from cubicweb.web.views.uicfg import reledit_ctrl
reledit_ctrl.tag_attribute(('Company', 'name'),
                           {'reload': lambda x:x.eid,
                            'default_value': xml_escape(u'<logilab tastes better>')})
reledit_ctrl.tag_object_of(('*', 'boss', 'Person'), {'edit_target': 'related'})

The default_value needs to be an xml escaped unicode string.

The edit_target tag on the boss relation being set to related will ensure edition of the Person entity instead (using a standard automatic form) of the association of Company and Person.

Finally, the reload key accepts either a boolean, an eid or a unicode string representing a url. If an eid is provided, it will be internally transformed into a url. The eid/url case helps when one needs to reload and the current url is inappropriate. A common case is edition of a key attribute, which is part of the current url. If one user changed the Company’s name from lozilab to logilab, reloading on http://myapp/company/lozilab would fail. Providing the entity’s eid, then, forces to reload on something like http://myapp/company/42, which always work.

Disable reledit#

By default, reledit is available on attributes and relations displayed in the ‘attribute’ section of the default primary view. If you want to disable it for some attribute or relation, you have use uicfg:

from cubicweb.web.views.uicfg import primaryview_display_ctrl as _pvdc
_pvdc.tag_attribute(('Company', 'name'), {'vid': 'incontext'})

To deactivate it everywhere it’s used automatically, you may use the code snippet below somewhere in your cube’s views:

from cubicweb.web.views import reledit

class DeactivatedAutoClickAndEditFormView(reledit.AutoClickAndEditFormView):
    def _should_edit_attribute(self, rschema):
        return False

    def _should_edit_attribute(self, rschema, role):
        return False

def registration_callback(vreg):
    vreg.register_and_replace(DeactivatedAutoClickAndEditFormView,
                              reledit.AutoClickAndEditFormView)

Base views#

CubicWeb provides a lot of standard views, that can be found in cubicweb.web.views sub-modules.

A certain number of views are used to build the web interface, which apply to one or more entities. As other appobjects, their identifier is what distinguish them from each others. The most generic ones, found in cubicweb.web.views.baseviews, are described below.

You’ll probably want to customize one or more of the described views which are default, generic, implementations.

You will also find modules providing some specific services:

Startup views#

Startup views are views requiring no context, from which you usually start browsing (for instance the index page). The usual selectors are none_rset or yes.

You’ll find here a description of startup views provided by the framework.

Other startup views:

schema

A view dedicated to the display of the schema of the instance

Boxes#

(cubicweb.web.views.boxes)

sidebox

This view displays usually a side box of some related entities in a primary view.

The action box#

The add_related is an automatic menu in the action box that allows to create an entity automatically related to the initial entity (context in which the box is displayed). By default, the links generated in this box are computed from the schema properties of the displayed entity, but it is possible to explicitly specify them thanks to the cubicweb.web.views.uicfg.rmode relation tag:

  • link, indicates that a relation is in general created pointing to an existing entity and that we should not to display a link for this relation

  • create, indicates that a relation is in general created pointing to new entities and that we should display a link to create a new entity and link to it automatically

If necessary, it is possible to overwrite the method relation_mode(rtype, targettype, x=’subject’) to dynamically compute a relation creation category.

Please note that if at least one action belongs to the addrelated category, the automatic behavior is desactivated in favor of an explicit behavior (e.g. display of addrelated category actions only).

Table views#

Example#

Let us take an example from the timesheet cube:

class ActivityResourcesTable(EntityView):
    __regid__ = 'activity.resources.table'
    __select__ = is_instance('Activity')

    def call(self, showresource=True):
        eids = ','.join(str(row[0]) for row in self.cw_rset)
        rql = ('Any R,D,DUR,WO,DESCR,S,A, SN,RT,WT ORDERBY D DESC '
               'WHERE '
               '   A is Activity, A done_by R, R title RT, '
               '   A diem D, A duration DUR, '
               '   A done_for WO, WO title WT, '
               '   A description DESCR, A in_state S, S name SN, '
               '   A eid IN (%s)' % eids)
        rset = self._cw.execute(rql)
        self.wview('resource.table', rset, 'null')

class ResourcesTable(RsetTableView):
    __regid__ = 'resource.table'
    # notice you may wish a stricter selector to check rql's shape
    __select__ = is_instance('Resource')
    # my table headers
    headers  = ['Resource', 'diem', 'duration', 'workpackage', 'description', 'state']
    # I want a table where attributes are editable (reledit inside)
    finalvid = 'editable-final'

    cellvids = {3: 'editable-final'}
    # display facets and actions with a menu
    layout_args = {'display_filter': 'top',
                   'add_view_actions': None}

To obtain an editable table, you may specify the ‘editable-table’ view identifier using some of cellvids, finalvid or nonfinalvid.

The previous example results in:

_images/views-table-shadow.png

In order to activate table filter mechanism, the display_filter option is given as a layout argument. A small arrow will be displayed at the table’s top right corner. Clicking on show filter form action, will display the filter form as below:

_images/views-table-filter-shadow.png

By the same way, you can display additional actions for the selected entities by setting add_view_actions layout option to True. This will add actions returned by the view’s table_actions().

You can notice that all columns of the result set are not displayed. This is because of given headers, implying to display only columns from 0 to len(headers).

Also Notice that the ResourcesTable view relies on a particular rql shape (which is not ensured by the way, the only checked thing is that the result set contains instance of the Resource type). That usually implies that you can’t use this view for user specific queries (e.g. generated by facets or typed manually).

So another option would be to write this view using EntityTableView, as below.

class ResourcesTable(EntityTableView):
    __regid__ = 'resource.table'
    __select__ = is_instance('Resource')
    # table columns definition
    columns  = ['resource', 'diem', 'duration', 'workpackage', 'description', 'in_state']
    # I want a table where attributes are editable (reledit inside)
    finalvid = 'editable-final'
    # display facets and actions with a menu
    layout_args = {'display_filter': 'top',
                   'add_view_actions': None}

    def workpackage_cell(entity):
        activity = entity.reverse_done_in[0]
        activity.view('reledit', rtype='done_for', role='subject', w=w)
    def workpackage_sortvalue(entity):
        activity = entity.reverse_done_in[0]
        return activity.done_for[0].sortvalue()

    column_renderers = {
        'resource': MainEntityColRenderer(),
        'workpackage': EntityTableColRenderer(
           header='Workpackage',
           renderfunc=workpackage_cell,
           sortfunc=workpackage_sortvalue,),
        'in_state': EntityTableColRenderer(
           renderfunc=lambda w,x: w(x.cw_adapt_to('IWorkflowable').printable_state),
           sortfunc=lambda x: x.cw_adapt_to('IWorkflowable').printable_state),
     }

Notice the following point:

  • cell_<column>(w, entity) will be searched for rendering the content of a cell. If not found, column is expected to be an attribute of entity.

  • cell_sortvalue_<column>(entity) should return a typed value to use for javascript sorting or None for not sortable columns (the default).

  • The etable_entity_sortvalue() decorator will set a ‘sortvalue’ function for the column containing the main entity (the one given as argument to all methods), which will call entity.sortvalue().

  • You can set a column header using the etable_header_title() decorator. This header will be translated. If it’s not an already existing msgid, think to mark it using _() (the example supposes headers are schema defined msgid).

Pro/cons of each approach#

EntityTableView and RsetableView provides basically the same set of features, though they don’t share the same properties. Let’s try to sum up pro and cons of each class.

  • EntityTableView view is:

    • more verbose, but usually easier to understand

    • easily extended (easy to add/remove columns for instance)

    • doesn’t rely on a particular rset shape. Simply give it a title and will be listed in the ‘possible views’ box if any.

  • RsetTableView view is:

    • hard to beat to display barely a result set, or for cases where some of headers, displaycols or cellvids could be defined to enhance the table while you don’t care about e.g. pagination or facets.

    • hardly extensible, as you usually have to change places where the view is called to modify the RQL (hence the view’s result set shape).

XML and RSS views#

(cubicweb.web.views.xmlrss)

Overview#

rss

Creates a RSS/XML view and call the view rssitem for each entity of the result set.

rssitem

Create a RSS/XML view for each entity based on the results of the dublin core methods of the entity (dc_*)

RSS Channel Example#

Assuming you have several blog entries, click on the title of the search box in the left column. A larger search box should appear. Enter:

Any X ORDERBY D WHERE X is BlogEntry, X creation_date D

and you get a list of blog entries.

Click on your login at the top right corner. Chose “user preferences”, then “boxes”, then “possible views box” and check “visible = yes” before validating your changes.

Enter the same query in the search box and you will see the same list, plus a box titled “possible views” in the left column. Click on “entityview”, then “RSS”.

You just applied the “RSS” view to the RQL selection you requested.

That’s it, you have a RSS channel for your blog.

Try again with:

Any X ORDERBY D WHERE X is BlogEntry, X creation_date D,
X entry_of B, B title "MyLife"

Another RSS channel, but a bit more focused.

A last one for the road:

Any C ORDERBY D WHERE C is Comment, C creation_date D LIMIT 15

displayed with the RSS view, that’s a channel for the last fifteen comments posted.

[WRITE ME]

  • show that the RSS view can be used to display an ordered selection of blog entries, thus providing a RSS channel

  • show that a different selection (by category) means a different channel

URL publishing#

(cubicweb.web.views.urlpublishing)

You can write your own URLPathEvaluator class to handle custom paths. For instance, if you want /my-card-id to redirect to the corresponding card’s primary view, you would write:

class CardWikiidEvaluator(URLPathEvaluator):
    priority = 3 # make it be evaluated *before* RestPathEvaluator

    def evaluate_path(self, req, segments):
        if len(segments) != 1:
            raise PathDontMatch()
        rset = req.execute('Any C WHERE C wikiid %(w)s',
                           {'w': segments[0]})
        if len(rset) == 0:
            # Raise NotFound if no card is found
            raise PathDontMatch()
        return None, rset

On the other hand, you can also deactivate some of the standard evaluators in your final application. The only thing you have to do is to unregister them, for instance in a registration_callback in your cube:

def registration_callback(vreg):
    vreg.unregister(RestPathEvaluator)

You can even replace the cubicweb.web.views.urlpublishing.URLPublisherComponent class if you want to customize the whole toolchain process or if you want to plug into an early enough extension point to control your request parameters:

class SanitizerPublisherComponent(URLPublisherComponent):
    """override default publisher component to explicitly ignore
    unauthorized request parameters in anonymous mode.
    """
    unauthorized_form_params = ('rql', 'vid', '__login', '__password')

    def process(self, req, path):
        if req.session.anonymous_session:
            self._remove_unauthorized_params(req)
        return super(SanitizerPublisherComponent, self).process(req, path)

    def _remove_unauthorized_params(self, req):
        for param in req.form.keys():
            if param in self.unauthorized_form_params:
                 req.form.pop(param)


def registration_callback(vreg):
    vreg.register_and_replace(SanitizerPublisherComponent, URLPublisherComponent)

URL rewriting#

(cubicweb.web.views.urlrewrite)

SimpleReqRewriter is enough for a certain number of simple cases. If it is not sufficient, SchemaBasedRewriter allows to do more elaborate things.

Here is an example of SimpleReqRewriter usage with plain string:

from cubicweb.web.views.urlrewrite import SimpleReqRewriter
class TrackerSimpleReqRewriter(SimpleReqRewriter):
    rules = [
     ('/versions', dict(vid='versionsinfo')),
     ]

When the url is <base_url>/versions, the view with the __regid__ versionsinfo is displayed.

Here is an example of SimpleReqRewriter usage with regular expressions:

from cubicweb.web.views.urlrewrite import (
    SimpleReqRewriter, rgx)

class BlogReqRewriter(SimpleReqRewriter):
    rules = [
        (rgx('/blogentry/([a-z_]+)\.rss'),
         dict(rql=('Any X ORDERBY CD DESC LIMIT 20 WHERE X is BlogEntry,'
                   'X creation_date CD, X created_by U, '
                   'U login "%(user)s"'
                   % {'user': r'\1'}), vid='rss'))
        ]

When a url matches the regular expression, the view with the __regid__ rss which match the result set is displayed.

To deal with URL rewriting with an underlying RQL query, it is possible to specify the behaviour in the case of an empty rset with the option empty_rset_raises_404.

The following example shows a SimpleReqRewriter usage with the empty_rset_raises_404 option set to True. In this case, the path mycwetypeurl/pouet will return a 404. Without this option, it would return a 200.

from cubicweb.web.views.urlrewrite import (
    SimpleReqRewriter, rgx)

class MyRewriter(SimpleReqRewriter):
    rules = [(rgx(r'/mycwetypeurl/([^/]+)'),
                                dict(vid='primary',
                                     rql=r'Any T WHERE T is CWEType, T name "\1"',
                                     empty_rset_raises_404=True),)]

Here is an example of SchemaBasedRewriter usage:

from cubicweb.web.views.urlrewrite import (
    SchemaBasedRewriter, rgx, build_rset)

class TrackerURLRewriter(SchemaBasedRewriter):
    rules = [
        (rgx('/project/([^/]+)/([^/]+)/tests'),
         build_rset(rql='Version X WHERE X version_of P, P name %(project)s, X num %(num)s',
                    rgxgroups=[('project', 1), ('num', 2)], vid='versiontests')),
        ]

The ‘download’ views#

Components#

Download views#

Embedded views#

Online documentation system#

Help views#

Actions#

Configuring the user interface#

Relation tags#

A RelationTag object is an object which allows to link a configuration information to a relation definition. For instance, the standard primary view uses a RelationTag object (uicfg.primaryview_section) to get the section to display relations.

# display ``entry_of`` relations in the ``relations`` section in the ``BlogEntry`` primary view
uicfg.primaryview_section.tag_subject_of(('BlogEntry', 'entry_of', '*'),
                                          'relations')

# hide every relation ``entry_of`` in the ``Blog`` primary view
uicfg.primaryview_section.tag_object_of(('*', 'entry_of', 'Blog'), 'hidden')
Three primitives are defined:

  • tag_subject_of tag a relation in the subject’s context

  • tag_object_of tag a relation in the object’s context

  • tag_attribute shortcut for tag_subject_of

The uicfg module#

Note

The part of uicfg that deals with primary views is in the Primary view configuration chapter.

The uihelper module#

Ajax#

Warning

This approach is deprecated in favor of using cwclientlibjs. If your use react for your UI, try the react components from the cwelements library. The documentation is kept here as reference.

For historical reference of what Ajax is and used to be, one can read the wikipedia article about Ajax.

CubicWeb provides a few helpers to facilitate javascript <-> python communications.

You can, for instance, register some python functions that will become callable from javascript through ajax calls. All the ajax URLs are handled by the cubicweb.web.views.ajaxcontroller.AjaxController controller.

Javascript#

CubicWeb uses quite a bit of javascript in its user interface and ships with jquery (1.3.x) and parts of the jquery UI library, plus a number of homegrown files and also other third party libraries.

All javascript files are stored in cubicweb/web/data/. There are around thirty js files there. In a cube it goes to data/.

Obviously one does not want javascript pieces to be loaded all at once, hence the framework provides a number of mechanisms and conventions to deal with javascript resources.

Conventions#

It is good practice to name cube specific js files after the name of the cube, like this : ‘cube.mycube.js’, so as to avoid name clashes.

Server-side Javascript API#

Javascript resources are typically loaded on demand, from views. The request object (available as self._cw from most application objects, for instance views and entities objects) has a few methods to do that:

  • add_js(self, jsfiles, localfile=True) which takes a sequence of javascript files and writes proper entries into the HTML header section. The localfile parameter allows to declare resources which are not from web/data (for instance, residing on a content delivery network).

  • add_onload(self, jscode) which adds one raw javascript code snippet inline in the html headers. This is quite useful for setting up early jQuery(document).ready(…) initialisations.

Javascript events#

  • server-response: this event is triggered on HTTP responses (both standard and ajax). The two following extra parameters are passed to callbacks :

    • ajax: a boolean that says if the reponse was issued by an ajax request

    • node: the DOM node returned by the server in case of an ajax request, otherwise the document itself for standard HTTP requests.

Important javascript AJAX APIS#

  • asyncRemoteExec and remoteExec are the base building blocks for doing arbitrary async (resp. sync) communications with the server

  • reloadComponent is a convenience function to replace a DOM node with server supplied content coming from a specific registry (this is quite handy to refresh the content of some boxes for instances)

  • jQuery.fn.loadxhtml is an important extension to jQuery which allows proper loading and in-place DOM update of xhtml views. It is suitably augmented to trigger necessary events, and process CubicWeb specific elements such as the facet system, fckeditor, etc.

A simple example with asyncRemoteExec#

On the python side, we have to define an cubicweb.web.views.ajaxcontroller.AjaxFunction object. The simplest way to do that is to use the cubicweb.web.views.ajaxcontroller.ajaxfunc() decorator (for more details on this, refer to Ajax).

On the javascript side, we do the asynchronous call. Notice how it creates a deferred object. Proper treatment of the return value or error handling has to be done through the addCallback and addErrback methods.

Anatomy of a reloadComponent call#

reloadComponent allows to dynamically replace some DOM node with new elements. It has the following signature:

  • compid (mandatory) is the name of the component to be reloaded

  • rql (optional) will be used to generate a result set given as argument to the selected component

  • registry (optional) defaults to ‘components’ but can be any other valid registry name

  • nodeid (optional) defaults to compid + ‘Component’ but can be any explicitly specified DOM node id

  • extraargs (optional) should be a dictionary of values that will be given to the cell_call method of the component

A simple reloadComponent example#

The server side implementation of reloadComponent is the cubicweb.web.views.ajaxcontroller.component() AjaxFunction appobject.

The following function implements a two-steps method to delete a standard bookmark and refresh the UI, while keeping the UI responsive.

function removeBookmark(beid) {
    d = asyncRemoteExec('delete_bookmark', beid);
    d.addCallback(function(boxcontent) {
        reloadComponent('bookmarks_box', '', 'boxes', 'bookmarks_box');
        document.location.hash = '#header';
        updateMessage(_("bookmark has been removed"));
     });
}

reloadComponent is called with the id of the bookmark box as argument, no rql expression (because the bookmarks display is actually independant of any dataset context), a reference to the ‘boxes’ registry (which hosts all left, right and contextual boxes) and finally an explicit ‘bookmarks_box’ nodeid argument that stipulates the target DOM node.

Anatomy of a loadxhtml call#

jQuery.fn.loadxhtml is an important extension to jQuery which allows proper loading and in-place DOM update of xhtml views. The existing jQuery.load function does not handle xhtml, hence the addition. The API of loadxhtml is roughly similar to that of jQuery.load.

  • url (mandatory) should be a complete url (typically referencing the cubicweb.web.views.ajaxcontroller.AjaxController, but this is not strictly mandatory)

  • data (optional) is a dictionary of values given to the controller specified through an url argument; some keys may have a special meaning depending on the choosen controller (such as fname for the JSonController); the callback key, if present, must refer to a function to be called at the end of loadxhtml (more on this below)

  • reqtype (optional) specifies the request method to be used (get or post); if the argument is ‘post’, then the post method is used, otherwise the get method is used

  • mode (optional) is one of replace (the default) which means the loaded node will replace the current node content, swap to replace the current node with the loaded node, and append which will append the loaded node to the current node content

About the callback option:

  • it is called with two parameters: the current node, and a list containing the loaded (and post-processed node)

  • whenever it returns another function, this function is called in turn with the same parameters as above

This mechanism allows callback chaining.

A simple example with loadxhtml#

Here we are concerned with the retrieval of a specific view to be injected in the live DOM. The view will be of course selected server-side using an entity eid provided by the client side.

from cubicweb.web.views.ajaxcontroller import ajaxfunc

@ajaxfunc(output_type='xhtml')
def frob_status(self, eid, frobname):
    entity = self._cw.entity_from_eid(eid)
    return entity.view('frob', name=frobname)

function updateSomeDiv(divid, eid, frobname) {
    var params = {fname:'frob_status', eid: eid, frobname:frobname};
    jQuery('#'+divid).loadxhtml(JSON_BASE_URL, params, 'post');
 }

In this example, the url argument is the base json url of a cube instance (it should contain something like http://myinstance/ajax?). The actual AjaxController method name is encoded in the params dictionary using the fname key.

A more real-life example#

A frequent need of Web 2 applications is the delayed (or demand driven) loading of pieces of the DOM. This is typically achieved using some preparation of the initial DOM nodes, jQuery event handling and proper use of loadxhtml.

We present here a skeletal version of the mecanism used in CubicWeb and available in web/views/tabs.py, in the LazyViewMixin class.

def lazyview(self, vid, rql=None):
    """ a lazy version of wview """
    self._cw.add_js('cubicweb.lazy.js')
    urlparams = {'vid' : vid, 'fname' : 'view'}
    if rql is not None:
        urlparams['rql'] = rql
    self.w(u'<div id="lazy-%s" cubicweb:loadurl="%s">',
           vid, xml_escape(self._cw.build_url('json', **urlparams)))
    self.w(u'</div>')
    self._cw.add_onload(u"""
        jQuery('#lazy-%(vid)s').bind('%(event)s', function() {
               loadNow('#lazy-%(vid)s');});"""
        % {'event': 'load_%s' % vid, 'vid': vid})

This creates a div with a specific event associated to it.

The full version deals with:

  • optional parameters such as an entity eid, an rset

  • the ability to further reload the fragment

  • the ability to display a spinning wheel while the fragment is still not loaded

  • handling of browsers that do not support ajax (search engines, text-based browsers such as lynx, etc.)

The javascript side is quite simple, due to loadxhtml awesomeness.

function loadNow(eltsel) {
    var lazydiv = jQuery(eltsel);
    lazydiv.loadxhtml(lazydiv.attr('cubicweb:loadurl'));
}

This is all significantly different of the previous simple example (albeit this example actually comes from real-life code).

Notice how the cubicweb:loadurl is used to convey the url information. The base of this url is similar to the global javascript JSON_BASE_URL. According to the pattern described earlier, the fname parameter refers to the standard js_view method of the JSonController. This method renders an arbitrary view provided a view id (or vid) is provided, and most likely an rql expression yielding a result set against which a proper view instance will be selected.

The cubicweb:loadurl is one of the 29 attributes extensions to XHTML in a specific cubicweb namespace. It is a means to pass information without breaking HTML nor XHTML compliance and without resorting to ungodly hacks.

Given all this, it is easy to add a small nevertheless useful feature to force the loading of a lazy view (for instance, a very computation-intensive web page could be scinded into one fast-loading part and a delayed part).

On the server side, a simple call to a javascript function is sufficient.

def forceview(self, vid):
    """trigger an event that will force immediate loading of the view
    on dom readyness
    """
    self._cw.add_onload("triggerLoad('%s');" % vid)

The browser-side definition follows.

function triggerLoad(divid) {
    jQuery('#lazy-' + divd).trigger('load_' + divid);
}

Javascript library: overview#

  • jquery.* : jquery and jquery UI library

  • cubicweb.ajax.js : concentrates all ajax related facilities (it extends jQuery with the loahxhtml function, provides a handfull of high-level ajaxy operations like asyncRemoteExec, reloadComponent, replacePageChunk, getDomFromResponse)

  • cubicweb.python.js : adds a number of practical extension to stdanrd javascript objects (on Date, Array, String, some list and dictionary operations), and a pythonesque way to build classes. Defines a CubicWeb namespace.

  • cubicweb.htmlhelpers.js : a small bag of convenience functions used in various other cubicweb javascript resources (baseuri, progress cursor handling, popup login box, html2dom function, etc.)

  • cubicweb.widgets.js : provides a widget namespace and constructors and helpers for various widgets (mainly facets and timeline)

  • cubicweb.edition.js : used by edition forms

  • cubicweb.preferences.js : used by the preference form

  • cubicweb.facets.js : used by the facets mechanism

There is also javascript support for massmailing, gmap (google maps), fckcwconfig (fck editor), timeline, calendar, goa (CubicWeb over AppEngine), flot (charts drawing), tabs and bookmarks.

API#

Javascript API#

Testing javascript#

You with the cubicweb.qunit.QUnitTestCase can include standard Qunit tests inside the python unittest run . You simply have to define a new class that inherit from QUnitTestCase and register your javascript test file in the all_js_tests lclass attribut. This all_js_tests is a sequence a 3-tuple (<test_file, [<dependencies> ,] [<data_files>]):

The <test_file> should contains the qunit test. <dependencies> defines the list of javascript file that must be imported before the test script. Dependencies are included their definition order. <data_files> are additional files copied in the test directory. both <dependencies> and <data_files> are optionnal. jquery.js is preincluded in for all test.

from cubicweb.qunit import QUnitTestCase

class MyQUnitTest(QUnitTestCase):

    all_js_tests = (
        ("relative/path/to/my_simple_testcase.js",)
        ("relative/path/to/my_qunit_testcase.js",(
            "rel/path/to/dependency_1.js",
            "rel/path/to/dependency_2.js",)),
        ("relative/path/to/my_complexe_qunit_testcase.js",(
             "rel/path/to/dependency_1.js",
             "rel/path/to/dependency_2.js",
           ),(
             "rel/path/file_dependency.html",
             "path/file_dependency.json")
            ),
        )

CSS Stylesheet#

Conventions#

Extending / overriding existing styles#

We cannot modify the order in which the application is reading the CSS. In the case we want to create new CSS style, the best is to define it a in a new CSS located under myapp/data/ and use those new styles while writing customized views and templates.

If you want to modify an existing CSS styling property, you will have to use !important declaration to override the existing property. The application apply a higher priority on the default CSS and you can not change that. Customized CSS will not be read first.

CubicWeb stylesheets#

Edition control#

This chapter covers the editing capabilities of CubicWeb. It explains html Form construction, the Edit Controller and their interactions.

HTML form construction#

CubicWeb provides the somewhat usual form / field / widget / renderer abstraction to provide generic building blocks which will greatly help you in building forms properly integrated with CubicWeb (coherent display, error handling, etc…), while keeping things as flexible as possible.

A form basically only holds a set of fields, and has te be bound to a renderer which is responsible to layout them. Each field is bound to a widget that will be used to fill in value(s) for that field (at form generation time) and ‘decode’ (fetch and give a proper Python type to) values sent back by the browser.

The field should be used according to the type of what you want to edit. E.g. if you want to edit some date, you’ll have to use the cubicweb.web.formfields.DateField. Then you can choose among multiple widgets to edit it, for instance cubicweb.web.formwidgets.TextInput (a bare text field), DateTimePicker (a simple calendar) or even JQueryDatePicker (the JQuery calendar). You can of course also write your own widget.

Exploring the available forms#

A small excursion into a CubicWeb shell is the quickest way to discover available forms (or application objects in general).

>>> from pprint import pprint
>>> pprint( session.vreg['forms'] )
{'base': [<class 'cubicweb.web.views.forms.FieldsForm'>,
          <class 'cubicweb.web.views.forms.EntityFieldsForm'>],
 'changestate': [<class 'cubicweb.web.views.workflow.ChangeStateForm'>,
                 <class 'cubicweb_tracker.views.forms.VersionChangeStateForm'>],
 'composite': [<class 'cubicweb.web.views.forms.CompositeForm'>,
               <class 'cubicweb.web.views.forms.CompositeEntityForm'>],
 'deleteconf': [<class 'cubicweb.web.views.editforms.DeleteConfForm'>],
 'edition': [<class 'cubicweb.web.views.autoform.AutomaticEntityForm'>,
             <class 'cubicweb.web.views.workflow.TransitionEditionForm'>,
             <class 'cubicweb.web.views.workflow.StateEditionForm'>],
 'logform': [<class 'cubicweb.web.views.basetemplates.LogForm'>],
 'massmailing': [<class 'cubicweb.web.views.massmailing.MassMailingForm'>],
 'muledit': [<class 'cubicweb.web.views.editforms.TableEditForm'>]}

The two most important form families here (for all practical purposes) are base and edition. Most of the time one wants alterations of the AutomaticEntityForm to generate custom forms to handle edition of an entity.

The Automatic Entity Form#

Anatomy of a choices function#

Let’s have a look at the ticket_done_in_choices function given to the choices parameter of the relation tag that is applied to the (‘Ticket’, ‘done_in’, ‘*’) relation definition, as it is both typical and sophisticated enough. This is a code snippet from the tracker cube.

The Ticket entity type can be related to a Project and a Version, respectively through the concerns and done_in relations. When a user is about to edit a ticket, we want to fill the combo box for the done_in relation with values pertinent with respect to the context. The important context here is:

  • creation or modification (we cannot fetch values the same way in either case)

  • __linkto url parameter given in a creation context

from cubicweb.web import formfields

def ticket_done_in_choices(form, field):
    entity = form.edited_entity
    # first see if its specified by __linkto form parameters
    linkedto = form.linked_to[('done_in', 'subject')]
    if linkedto:
        return linkedto
    # it isn't, get initial values
    vocab = field.relvoc_init(form)
    veid = None
    # try to fetch the (already or pending) related version and project
    if not entity.has_eid():
        peids = form.linked_to[('concerns', 'subject')]
        peid = peids and peids[0]
    else:
        peid = entity.project.eid
        veid = entity.done_in and entity.done_in[0].eid
    if peid:
        # we can complete the vocabulary with relevant values
        rschema = form._cw.vreg.schema['done_in'].rdef('Ticket', 'Version')
        rset = form._cw.execute(
            'Any V, VN ORDERBY version_sort_value(VN) '
            'WHERE V version_of P, P eid %(p)s, V num VN, '
            'V in_state ST, NOT ST name "published"', {'p': peid}, 'p')
        vocab += [(v.view('combobox'), v.eid) for v in rset.entities()
                  if rschema.has_perm(form._cw, 'add', toeid=v.eid)
                  and v.eid != veid]
    return vocab

The first thing we have to do is fetch potential values from the __linkto url parameter that is often found in entity creation contexts (the creation action provides such a parameter with a predetermined value; for instance in this case, ticket creation could occur in the context of a Version entity). The RelationField field class provides a relvoc_linkedto() method that gets a list suitably filled with vocabulary values.

linkedto = field.relvoc_linkedto(form)
if linkedto:
    return linkedto

Then, if no __linkto argument was given, we must prepare the vocabulary with an initial empty value (because done_in is not mandatory, we must allow the user to not select a verson) and already linked values. This is done with the relvoc_init() method.

vocab = field.relvoc_init(form)

But then, we have to give more: if the ticket is related to a project, we should provide all the non published versions of this project (Version and Project can be related through the version_of relation). Conversely, if we do not know yet the project, it would not make sense to propose all existing versions as it could potentially lead to incoherences. Even if these will be caught by some RQLConstraint, it is wise not to tempt the user with error-inducing candidate values.

The “ticket is related to a project” part must be decomposed as:

  • this is a new ticket which is created is the context of a project

  • this is an already existing ticket, linked to a project (through the concerns relation)

  • there is no related project (quite unlikely given the cardinality of the concerns relation, so it can only mean that we are creating a new ticket, and a project is about to be selected but there is no __linkto argument)

Note

the last situation could happen in several ways, but of course in a polished application, the paths to ticket creation should be controlled so as to avoid a suboptimal end-user experience

Hence, we try to fetch the related project.

veid = None
if not entity.has_eid():
    peids = form.linked_to[('concerns', 'subject')]
    peid = peids and peids[0]
else:
    peid = entity.project.eid
    veid = entity.done_in and entity.done_in[0].eid

We distinguish between entity creation and entity modification using the Entity.has_eid() method, which returns False on creation. At creation time the only way to get a project is through the __linkto parameter. Notice that we fetch the version in which the ticket is done_in if any, for later.

Note

the implementation above assumes that if there is a __linkto parameter, it is only about a project. While it makes sense most of the time, it is not an absolute. Depending on how an entity creation action action url is built, several outcomes could be possible there

If the ticket is already linked to a project, fetching it is trivial. Then we add the relevant version to the initial vocabulary.

if peid:
    rschema = form._cw.vreg.schema['done_in'].rdef('Ticket', 'Version')
    rset = form._cw.execute(
        'Any V, VN ORDERBY version_sort_value(VN) '
        'WHERE V version_of P, P eid %(p)s, V num VN, '
        'V in_state ST, NOT ST name "published"', {'p': peid})
    vocab += [(v.view('combobox'), v.eid) for v in rset.entities()
              if rschema.has_perm(form._cw, 'add', toeid=v.eid)
              and v.eid != veid]

Warning

we have to defend ourselves against lack of a project eid. Given the cardinality of the concerns relation, there must be a project, but this rule can only be enforced at validation time, which will happen of course only after form subsmission

Here, given a project eid, we complete the vocabulary with all unpublished versions defined in the project (sorted by number) for which the current user is allowed to establish the relation.

Building self-posted form with custom fields/widgets#

Sometimes you want a form that is not related to entity edition. For those, you’ll have to handle form posting by yourself. Here is a complete example on how to achieve this (and more).

Imagine you want a form that selects a month period. There are no proper field/widget to handle this in CubicWeb, so let’s start by defining them:

# let's have the whole import list at the beginning, even those necessary for
# subsequent snippets
from logilab.common import date
from logilab.mtconverter import xml_escape
from cubicweb.view import View
from cubicweb.predicates import match_kwargs
from cubicweb.web import RequestError, ProcessFormError
from cubicweb.web import formfields as fields, formwidgets as wdgs
from cubicweb.web.views import forms, calendar

class MonthSelect(wdgs.Select):
    """Custom widget to display month and year. Expect value to be given as a
    date instance.
    """

    def format_value(self, form, field, value):
        return u'%s/%s' % (value.year, value.month)

    def process_field_data(self, form, field):
        val = super(MonthSelect, self).process_field_data(form, field)
        try:
            year, month = val.split('/')
            year = int(year)
            month = int(month)
            return date.date(year, month, 1)
        except ValueError:
            raise ProcessFormError(
                form._cw._('badly formated date string %s') % val)


class MonthPeriodField(fields.CompoundField):
    """custom field composed of two subfields, 'begin_month' and 'end_month'.

    It expects to be used on form that has 'mindate' and 'maxdate' in its
    extra arguments, telling the range of month to display.
    """

    def __init__(self, *args, **kwargs):
        kwargs.setdefault('widget', wdgs.IntervalWidget())
        super(MonthPeriodField, self).__init__(
            [fields.StringField(name='begin_month',
                                choices=self.get_range, sort=False,
                                value=self.get_mindate,
                                widget=MonthSelect()),
             fields.StringField(name='end_month',
                                choices=self.get_range, sort=False,
                                value=self.get_maxdate,
                                widget=MonthSelect())], *args, **kwargs)

    @staticmethod
    def get_range(form, field):
        mindate = date.todate(form.cw_extra_kwargs['mindate'])
        maxdate = date.todate(form.cw_extra_kwargs['maxdate'])
        assert mindate <= maxdate
        _ = form._cw._
        months = []
        while mindate <= maxdate:
            label = '%s %s' % (_(calendar.MONTHNAMES[mindate.month - 1]),
                               mindate.year)
            value = field.widget.format_value(form, field, mindate)
            months.append( (label, value) )
            mindate = date.next_month(mindate)
        return months

    @staticmethod
    def get_mindate(form, field):
        return form.cw_extra_kwargs['mindate']

    @staticmethod
    def get_maxdate(form, field):
        return form.cw_extra_kwargs['maxdate']

    def process_posted(self, form):
        for field, value in super(MonthPeriodField, self).process_posted(form):
            if field.name == 'end_month':
                value = date.last_day(value)
            yield field, value

Here we first define a widget that will be used to select the beginning and the end of the period, displaying months like ‘<month> YYYY’ but using ‘YYYY/mm’ as actual value.

We then define a field that will actually hold two fields, one for the beginning and another for the end of the period. Each subfield uses the widget we defined earlier, and the outer field itself uses the standard IntervalWidget. The field adds some logic:

  • a vocabulary generation function get_range, used to populate each sub-field

  • two ‘value’ functions get_mindate and get_maxdate, used to tell to subfields which value they should consider on form initialization

  • overriding of process_posted, called when the form is being posted, so that the end of the period is properly set to the last day of the month.

Now, we can define a very simple form:

class MonthPeriodSelectorForm(forms.FieldsForm):
    __regid__ = 'myform'
    __select__ = match_kwargs('mindate', 'maxdate')

    form_buttons = [wdgs.SubmitButton()]
    form_renderer_id = 'onerowtable'
    period = MonthPeriodField()

where we simply add our field, set a submit button and use a very simple renderer (try others!). Also we specify a selector that ensures form will have arguments necessary to our field.

Now, we need a view that will wrap the form and handle post when it occurs, simply displaying posted values in the page:

class SelfPostingForm(View):
    __regid__ = 'myformview'

    def call(self):
        mindate, maxdate = date.date(2010, 1, 1), date.date(2012, 1, 1)
        form = self._cw.vreg['forms'].select(
            'myform', self._cw, mindate=mindate, maxdate=maxdate, action='')
        try:
            posted = form.process_posted()
            self.w(u'<p>posted values %s</p>' % xml_escape(repr(posted)))
        except RequestError: # no specified period asked
            pass
        form.render(w=self.w, formvalues=self._cw.form)

Notice usage of the process_posted() method, that will return a dictionary of typed values (because they have been processed by the field). In our case, when the form is posted you should see a dictionary with ‘begin_month’ and ‘end_month’ as keys with the selected dates as value (as a python date object).

APIs#

Dissection of an entity form#

This is done (again) with a vanilla instance of the tracker cube. We will populate the database with a bunch of entities and see what kind of job the automatic entity form does.

Populating the database#

We should start by setting up a bit of context: a project with two unpublished versions, and a ticket linked to the project and the first version.

>>> p = rql('INSERT Project P: P name "cubicweb"')
>>> for num in ('0.1.0', '0.2.0'):
...  rql('INSERT Version V: V num "%s", V version_of P WHERE P eid %%(p)s' % num, {'p': p[0][0]})
...
<resultset 'INSERT Version V: V num "0.1.0", V version_of P WHERE P eid %(p)s' (1 rows): [765L] (('Version',))>
<resultset 'INSERT Version V: V num "0.2.0", V version_of P WHERE P eid %(p)s' (1 rows): [766L] (('Version',))>
>>> t = rql('INSERT Ticket T: T title "let us write more doc", T done_in V, '
            'T concerns P WHERE V num "0.1.0"', P eid %(p)s', {'p': p[0][0]})
>>> commit()

Now let’s see what the edition form builds for us.

>>> cnx.use_web_compatible_requests('http://fakeurl.com')
>>> req = cnx.request()
>>> form = req.vreg['forms'].select('edition', req, rset=rql('Ticket T'))
>>> html = form.render()

Note

In order to play interactively with web side application objects, we have to cheat a bit to have request object that will looks like HTTP request object, by calling use_web_compatible_requests() on the connection.

This creates an automatic entity form. The .render() call yields an html (unicode) string. The html output is shown below (with internal fieldset omitted).

Looking at the html output#

The form enveloppe#
<div class="iformTitle"><span>main informations</span></div>
<div class="formBody">
 <form action="http://crater:9999/validateform" method="post" enctype="application/x-www-form-urlencoded"
       id="entityForm" onsubmit="return freezeFormButtons(&#39;entityForm&#39;);"
       class="entityForm" target="eformframe">
   <div id="progress">validating...</div>
   <fieldset>
     <input name="__form_id" type="hidden" value="edition" />
     <input name="__errorurl" type="hidden" value="http://perdu.com#entityForm" />
     <input name="__domid" type="hidden" value="entityForm" />
     <input name="__type:763" type="hidden" value="Ticket" />
     <input name="eid" type="hidden" value="763" />
     <input name="__maineid" type="hidden" value="763" />
     <input name="_cw_edited_fields:763" type="hidden"
            value="concerns-subject,done_in-subject,priority-subject,type-subject,title-subject,description-subject,__type,_cw_generic_field" />
     ...
   </fieldset>
   <iframe width="0px" height="0px" name="eformframe" id="eformframe" src="javascript: void(0);"></iframe>
  </form>
</div>

The main fieldset encloses a set of hidden fields containing various metadata, that will be used by the edit controller to process it back correctly.

The freezeFormButtons(…) javascript callback defined on the onlick event of the form element prevents accidental multiple clicks in a row.

The action of the form is mapped to the validateform controller (situated in cubicweb.web.views.basecontrollers).

A full explanation of the validation loop is given in The form validation process.

The attributes section#

We can have a look at some of the inner nodes of the form. Some fields are omitted as they are redundant for our purposes.

<fieldset class="default">
  <table class="attributeForm">
    <tr class="title_subject_row">
      <th class="labelCol"><label class="required" for="title-subject:763">title</label></th>
      <td>
        <input id="title-subject:763" maxlength="128" name="title-subject:763" size="45"
               type="text" value="let us write more doc" />
      </td>
    </tr>
    ... (description field omitted) ...
    <tr class="priority_subject_row">
      <th class="labelCol"><label class="required" for="priority-subject:763">priority</label></th>
      <td>
        <select id="priority-subject:763" name="priority-subject:763" size="1">
          <option value="important">important</option>
          <option selected="selected" value="normal">normal</option>
          <option value="minor">minor</option>
        </select>
        <div class="helper">importance</div>
      </td>
    </tr>
    ... (type field omitted) ...
    <tr class="concerns_subject_row">
      <th class="labelCol"><label class="required" for="concerns-subject:763">concerns</label></th>
      <td>
        <select id="concerns-subject:763" name="concerns-subject:763" size="1">
          <option selected="selected" value="760">Foo</option>
        </select>
      </td>
    </tr>
    <tr class="done_in_subject_row">
      <th class="labelCol"><label for="done_in-subject:763">done in</label></th>
      <td>
        <select id="done_in-subject:763" name="done_in-subject:763" size="1">
          <option value="__cubicweb_internal_field__"></option>
          <option selected="selected" value="761">Foo 0.1.0</option>
          <option value="762">Foo 0.2.0</option>
        </select>
        <div class="helper">version in which this ticket will be / has been  done</div>
      </td>
    </tr>
  </table>
</fieldset>

Note that the whole form layout has been computed by the form renderer. It is the renderer which produces the table structure. Otherwise, the fields html structure is emitted by their associated widget.

While it is called the attributes section of the form, it actually contains attributes and mandatory relations. For each field, we observe:

  • a dedicated row with a specific class, such as title_subject_row (responsability of the form renderer)

  • an html widget (input, select, …) with:

    • an id built from the rtype-role:eid pattern

    • a name built from the same pattern

    • possible values or preselected options

The relations section#
<fieldset class="This ticket :">
  <legend>This ticket :</legend>
  <table class="attributeForm">
    <tr class="_cw_generic_field_None_row">
      <td colspan="2">
        <table id="relatedEntities">
          <tr><th>&#160;</th><td>&#160;</td></tr>
          <tr id="relationSelectorRow_763" class="separator">
            <th class="labelCol">
              <select id="relationSelector_763"
                      onchange="javascript:showMatchingSelect(this.options[this.selectedIndex].value,763);">
                <option value="">select a relation</option>
                <option value="appeared_in_subject">appeared in</option>
                <option value="custom_workflow_subject">custom workflow</option>
                <option value="depends_on_object">dependency of</option>
                <option value="depends_on_subject">depends on</option>
                <option value="identical_to_subject">identical to</option>
                <option value="see_also_subject">see also</option>
              </select>
            </th>
            <td id="unrelatedDivs_763"></td>
          </tr>
        </table>
      </td>
    </tr>
  </table>
</fieldset>

The optional relations are grouped into a drop-down combo box. Selection of an item triggers a javascript function which will:

  • show already related entities in the div of id relatedentities using a two-colown layout, with an action to allow deletion of individual relations (there are none in this example)

  • provide a relation selector in the div of id relationSelector_EID to allow the user to set up relations and trigger dynamic action on the last div

  • fill the div of id unrelatedDivs_EID with a dynamically computed selection widget allowing direct selection of an unrelated (but relatable) entity or a switch towards the search mode of CubicWeb which allows full browsing and selection of an entity using a dedicated action situated in the left column boxes.

The buttons zone#

Finally comes the buttons zone.

<table width="100%">
  <tbody>
    <tr>
      <td align="center">
        <button class="validateButton" type="submit" value="validate">
          <img alt="OK_ICON" src="http://myapp/datafd8b5d92771209ede1018a8d5da46a37/ok.png" />
          validate
        </button>
      </td>
      <td style="align: right; width: 50%;">
        <button class="validateButton"
                onclick="postForm(&#39;__action_apply&#39;, &#39;button_apply&#39;, &#39;entityForm&#39;)"
                type="button" value="apply">
          <img alt="APPLY_ICON" src="http://myapp/datafd8b5d92771209ede1018a8d5da46a37/plus.png" />
          apply
        </button>
        <button class="validateButton"
                onclick="postForm(&#39;__action_cancel&#39;, &#39;button_cancel&#39;, &#39;entityForm&#39;)"
                type="button" value="cancel">
          <img alt="CANCEL_ICON" src="http://myapp/datafd8b5d92771209ede1018a8d5da46a37/cancel.png" />
          cancel
        </button>
      </td>
    </tr>
  </tbody>
</table>

The most notable artifacts here are the postForm(...) calls defined on click events on these buttons. This function basically submits the form.

The form validation process#

Validation loop#

On form submission, the form.action is invoked. Basically, the validateform controller is called and its output lands in the specified target, an invisible <iframe> at the end of the form.

Hence, the main page is not replaced, only the iframe contents. The validateform controller only outputs a tiny javascript fragment which is then immediately executed.

<iframe width="0px" height="0px" name="eformframe" id="eformframe" src="javascript: void(0);">
  <script type="text/javascript">
    window.parent.handleFormValidationResponse('entityForm', null, null,
                                               [false, [2164, {"name-subject": "required field"}], null],
                                               null);
  </script>
</iframe>

The window.parent part ensures the javascript function is called on the right context (that is: the form element). We will describe its parameters:

  • first comes the form id (entityForm)

  • then two optional callbacks for the success and failure case

  • an array containing:

    • a boolean which indicates status (success or failure), and then, on error:

      • an array structured as [eid, {'rtype-role': 'error msg'}, ...]

    • on success:

      • a url (string) representing the next thing to jump to

Given the array structure described above, it is quite simple to manipulate the DOM to show the errors at appropriate places.

Explanation#

This mecanism may seem a bit overcomplicated but we have to deal with two realities:

  • in the (strict) XHTML world, there are no iframes (hence the dynamic inclusion, tolerated by Firefox)

  • no (or not all) browser(s) support file input field handling through ajax.

The edit controller#

It can be found in (cubicweb.web.views.editcontroller). This controller processes data received from an html form to create or update entities.

Edition handling#

The parameters related to entities to edit are specified as follows (first seen in The attributes section):

<rtype-role>:<entity eid>

where entity eid could be a letter in case of an entity to create. We name those parameters as qualified.

  • Retrieval of entities to edit is done by using the forms parameters eid and __type

  • For all the attributes and the relations of an entity to edit (attributes and relations are handled a bit differently but these details are not much relevant here) :

    • using the rtype, role and __type information, fetch an appropriate field instance

    • check if the field has been modified (if not, proceed to the next relation)

    • build an rql expression to update the entity

At the end, all rql expressions are executed.

  • For each entity to edit:

    • if a qualified parameter __linkto is specified, its value has to be a string (or a list of strings) such as:

      <relation type>:<eids>:<target>

      where <target> is either subject or object and each eid could be separated from the others by a _. Target specifies if the edited entity is subject or object of the relation and each relation specified will be inserted.

    • if a qualified parameter __clone_eid is specified for an entity, the relations of the specified entity passed as value of this parameter are copied on the edited entity.

    • if a qualified parameter __delete is specified, its value must be a string or a list of string such as follows:

      <subjects eids>:<relation type>:<objects eids>

      where each eid subject or object can be seperated from the other by _. Each specified relation will be deleted.

  • If no entity is edited but the form contains the parameters __linkto and eid, this one is interpreted by using the value specified for eid to designate the entity on which to add the relations.

Note

  • if the parameter __action_delete is found, all the entities specified as to be edited will be deleted.

  • if the parameter __action_cancel is found, no action is completed.

  • if the parameter __action_apply is found, the editing is applied normally but the redirection is done on the form (see Redirection control).

  • if no entity is found to be edited and if there is no parameter __action_delete, __action_cancel, __linkto, __delete or __insert, an error is raised.

  • using the parameter __message in the form will allow to use its value as a message to provide the user once the editing is completed.

Redirection control#

Once editing is completed, there is still an issue left: where should we go now? If nothing is specified, the controller will do his job but it does not mean we will be happy with the result. We can control that by using the following parameters:

  • __redirectpath: path of the URL (relative to the root URL of the site, no form parameters

  • __redirectparams: forms parameters to add to the path

  • __redirectrql: redirection RQL request

  • __redirectvid: redirection view identifier

  • __errorurl: initial form URL, used for redirecting in case a validation error is raised during editing. If this one is not specified, an error page is displayed instead of going back to the form (which is, if necessary, responsible for displaying the errors)

  • __form_id: initial view form identifier, used if __action_apply is found

In general we use either __redirectpath and __redirectparams or __redirectrql and __redirectvid.

Examples#

(Automatic) Entity form#

Looking at some cubes available on the cubicweb forge we find some with form manipulation. The following example comes from the the conference cube. It extends the change state form for the case where a Talk entity is getting into submitted state. The goal is to select reviewers for the submitted talk.

from cubicweb.web import formfields as ff, formwidgets as fwdgs
class SendToReviewerStatusChangeView(ChangeStateFormView):
    __select__ = (ChangeStateFormView.__select__ &
                  is_instance('Talk') &
                  rql_condition('X in_state S, S name "submitted"'))

    def get_form(self, entity, transition, **kwargs):
        form = super(SendToReviewerStatusChangeView, self).get_form(entity, transition, **kwargs)
        relation = ff.RelationField(name='reviews', role='object',
                                    eidparam=True,
                                    label=_('select reviewers'),
                                    widget=fwdgs.Select(multiple=True))
        form.append_field(relation)
        return form

Simple extension of a form can be done from within the FormView wrapping the form. FormView instances have a handy get_form method that returns the form to be rendered. Here we add a RelationField to the base state change form.

One notable point is the eidparam argument: it tells both the field and the edit controller that the field is linked to a specific entity.

It is hence entirely possible to add ad-hoc fields that will be processed by some specialized instance of the edit controller.

Ad-hoc fields form#

We want to define a form doing something else than editing an entity. The idea is to propose a form to send an email to entities in a resultset which implements IEmailable. Let’s take a simplified version of what you’ll find in cubicweb.web.views.massmailing.

Here is the source code:

def sender_value(form, field):
    return '%s <%s>' % (form._cw.user.dc_title(), form._cw.user.get_email())

def recipient_choices(form, field):
    return [(e.get_email(), e.eid)
             for e in form.cw_rset.entities()
             if e.get_email()]

def recipient_value(form, field):
    return [e.eid for e in form.cw_rset.entities()
            if e.get_email()]

class MassMailingForm(forms.FieldsForm):
    __regid__ = 'massmailing'

    needs_js = ('cubicweb.widgets.js',)
    domid = 'sendmail'
    action = 'sendmail'

    sender = ff.StringField(widget=TextInput({'disabled': 'disabled'}),
                            label=_('From:'),
                            value=sender_value)

    recipient = ff.StringField(widget=CheckBox(),
                               label=_('Recipients:'),
                               choices=recipient_choices,
                               value=recipients_value)

    subject = ff.StringField(label=_('Subject:'), max_length=256)

    mailbody = ff.StringField(widget=AjaxWidget(wdgtype='TemplateTextField',
                                                inputid='mailbody'))

    form_buttons = [ImgButton('sendbutton', "javascript: $('#sendmail').submit()",
                              _('send email'), 'SEND_EMAIL_ICON'),
                    ImgButton('cancelbutton', "javascript: history.back()",
                              stdmsgs.BUTTON_CANCEL, 'CANCEL_EMAIL_ICON')]

Let’s detail what’s going on up there. Our form will hold four fields:

  • a sender field, which is disabled and will simply contains the user’s name and email

  • a recipients field, which will be displayed as a list of users in the context result set with checkboxes so user can still choose who will receive his mailing by checking or not the checkboxes. By default all of them will be checked since field’s value return a list containing same eids as those returned by the vocabulary function.

  • a subject field, limited to 256 characters (hence we know a TextInput will be used, as explained in StringField)

  • a mailbody field. This field use an ajax widget, defined in cubicweb.widgets.js, and whose definition won’t be shown here. Notice though that we tell this form need this javascript file by using needs_js

Last but not least, we add two buttons control: one to post the form using javascript ($(‘#sendmail’) being the jQuery call to get the element with DOM id set to ‘sendmail’, which is our form DOM id as specified by its domid attribute), another to cancel the form which will go back to the previous page using another javascript call. Also we specify an image to use as button icon as a resource identifier (see Step 1: tired of the default look?) given as last argument to cubicweb.web.formwidgets.ImgButton.

To see this form, we still have to wrap it in a view. This is pretty simple:

class MassMailingFormView(form.FormViewMixIn, EntityView):
    __regid__ = 'massmailing'
    __select__ = is_instance(IEmailable) & authenticated_user()

    def call(self):
        form = self._cw.vreg['forms'].select('massmailing', self._cw,
                                             rset=self.cw_rset)
        form.render(w=self.w)

As you see, we simply define a view with proper selector so it only apply to a result set containing IEmailable entities, and so that only users in the managers or users group can use it. Then in the call() method for this view we simply select the above form and call its .render() method with our output stream as argument.

When this form is submitted, a controller with id ‘sendmail’ will be called (as specified using action). This controller will be responsible to actually send the mail to specified recipients.

Here is what it looks like:

class SendMailController(Controller):
    __regid__ = 'sendmail'
    __select__ = (authenticated_user() &
                  match_form_params('recipient', 'mailbody', 'subject'))

    def publish(self, rset=None):
        body = self._cw.form['mailbody']
        subject = self._cw.form['subject']
        eids = self._cw.form['recipient']
        # eids may be a string if only one recipient was specified
        if isinstance(eids, basestring):
            rset = self._cw.execute('Any X WHERE X eid %(x)s', {'x': eids})
        else:
            rset = self._cw.execute('Any X WHERE X eid in (%s)' % (','.join(eids)))
        recipients = list(rset.entities())
        msg = format_mail({'email' : self._cw.user.get_email(),
                           'name' : self._cw.user.dc_title()},
                          recipients, body, subject)
        if not self._cw.vreg.config.sendmails([(msg, recipients)]):
            msg = self._cw._('could not connect to the SMTP server')
        else:
            msg = self._cw._('emails successfully sent')
        raise Redirect(self._cw.build_url(__message=msg))

The entry point of a controller is the publish method. In that case we simply get back post values in request’s form attribute, get user instances according to eids found in the ‘recipient’ form value, and send email after calling format_mail() to get a proper email message. If we can’t send email or if we successfully sent email, we redirect to the index page with proper message to inform the user.

Also notice that our controller has a selector that deny access to it to anonymous users (we don’t want our instance to be used as a spam relay), but also checks if the expected parameters are specified in forms. That avoids later defensive programming (though it’s not enough to handle all possible error cases).

To conclude our example, suppose we wish a different form layout and that existent renderers are not satisfying (we would check that first of course :). We would then have to define our own renderer:

class MassMailingFormRenderer(formrenderers.FormRenderer):
    __regid__ = 'massmailing'

    def _render_fields(self, fields, w, form):
        w(u'<table class="headersform">')
        for field in fields:
            if field.name == 'mailbody':
                w(u'</table>')
                w(u'<div id="toolbar">')
                w(u'<ul>')
                for button in form.form_buttons:
                    w(u'<li>%s</li>' % button.render(form))
                w(u'</ul>')
                w(u'</div>')
                w(u'<div>')
                w(field.render(form, self))
                w(u'</div>')
            else:
                w(u'<tr>')
                w(u'<td class="hlabel">%s</td>' %
                  self.render_label(form, field))
                w(u'<td class="hvalue">')
                w(field.render(form, self))
                w(u'</td></tr>')

    def render_buttons(self, w, form):
        pass

We simply override the _render_fields and render_buttons method of the base form renderer to arrange fields as we desire it: here we’ll have first a two columns table with label and value of the sender, recipients and subject field (form order respected), then form controls, then a div containing the textarea for the email’s content.

To bind this renderer to our form, we should add to our form definition above:

form_renderer_id = 'massmailing'

The facets system#

Facets allow to restrict searches according to some user friendly criterias. CubicWeb has a builtin facet system to define restrictions filters really as easily as possible.

Here is an exemple of the facets rendering picked from our http://www.cubicweb.org web site:

_images/facet_overview.png

Facets will appear on each page presenting more than one entity that may be filtered according to some known criteria.

Base classes for facets#

Internationalization#

Cubicweb fully supports the internalization of its content and interface.

Cubicweb’s interface internationalization is based on the translation project GNU gettext.

Cubicweb’ internalization involves two steps:

  • in your Python code and cubicweb-tal templates : mark translatable strings

  • in your instance : handle the translation catalog, edit translations

String internationalization#

User defined string#

In the Python code and cubicweb-tal templates translatable strings can be marked in one of the following ways :

  • by using the built-in function _:

    class PrimaryView(EntityView):
    """the full view of an non final entity"""
    __regid__ = 'primary'
    title = _('primary')

OR

  • by using the equivalent request’s method:

    class NoResultView(View):
    """default view when no result has been found"""
    __regid__ = 'noresult'

    def call(self, **kwargs):
        self.w(u'<div class="searchMessage"><strong>%s</strong></div>\n'
            % self._cw._('No result matching query'))

The goal of the built-in function _ is only to mark the translatable strings, it will only return the string to translate itself, but not its translation (it’s actually another name for the unicode builtin).

In the other hand the request’s method self._cw._ is also meant to retrieve the proper translation of translation strings in the requested language.

Finally you can also use the __ (two underscores) attribute of request object to get a translation for a string which should not itself added to the catalog, usually in case where the actual msgid is created by string interpolation

self._cw.__('This %s' % etype)

In this example ._cw.__ is used instead of ._cw._ so we don’t have ‘This %s’ in messages catalogs.

Translations in cubicweb-tal template can also be done with TAL tags i18n:content and i18n:replace.

If you need to mark other messages as translatable, you can create a file named i18n/static-messages.pot, see for example Specialize translation for an application cube.

You could put there messages not found in the python sources or overrides some messages that are in cubes used in the dependencies.

Generated string#

We do not need to mark the translation strings of entities/relations used by a particular instance’s schema as they are generated automatically. String for various actions are also generated.

For exemple the following schema:

class EntityA(EntityType):
    relation_a2b = SubjectRelation('EntityB')

class EntityB(EntityType):
    pass

May generate the following message

add EntityA relation_a2b EntityB subject

This message will be used in views of EntityA for creation of a new EntityB with a preset relation relation_a2b between the current EntityA and the new EntityB. The opposite message

add EntityA relation_a2b EntityB object

Is used for similar creation of an EntityA from a view of EntityB. The title of they respective creation form will be

creating EntityB (EntityA %(linkto)s relation_a2b EntityB)

creating EntityA (EntityA relation_a2b %(linkto)s EntityA)

In the translated string you can use %(linkto)s for reference to the source entity.

Handling the translation catalog#

Once the internationalization is done in your code, you need to populate and update the translation catalog. Cubicweb provides the following commands for this purpose:

  • i18ncubicweb updates Cubicweb framework’s translation catalogs. Unless you actually work on the framework itself, you don’t need to use this command.

  • i18ncube updates the translation catalogs of one particular cube (or of all cubes). After this command is executed you must update the translation files .po in the “i18n” directory of your cube. This command will of course not remove existing translations still in use. It will mark unused translation but not remove them.

  • i18ninstance recompiles the translation catalogs of one particular instance (or of all instances) after the translation catalogs of its cubes have been updated. This command is automatically called every time you create or update your instance. The compiled catalogs (.mo) are stored in the i18n/<lang>/LC_MESSAGES of instance where lang is the language identifier (‘en’ or ‘fr’ for exemple).

Example#

You have added and/or modified some translation strings in your cube (after creating a new view or modifying the cube’s schema for exemple). To update the translation catalogs you need to do:

  1. cubicweb-ctl i18ncube <cube>

  2. Edit the <cube>/i18n/xxx.po files and add missing translations (those with an empty msgstr)

  3. hg ci -m “updated i18n catalogs”

  4. cubicweb-ctl i18ninstance <myinstance>

Customizing the messages extraction process#

The messages extraction performed by the i18ncommand collects messages from a few different sources:

  • the schema and application definition (entity names, docstrings, help messages, uicfg),

  • the source files:

    • i18n:content or i18n:replace directives from TAL files (with .pt extension),

    • strings prefixed by an underscore (_) in python files,

    • strings with double quotes prefixed by an underscore in javascript files.

The source files are collected by walking through the cube directory, but ignoring a few directories like .hg, .tox, test or node_modules.

If you need to customize this behaviour in your cube, you have to extend the cubicweb.devtools.devctl.I18nCubeMessageExtractor. The example below will collect strings from jinja2 files and ignore the static directory during the messages collection phase:

# mymodule.py
from cubicweb.devtools import devctl

class MyMessageExtractor(devctl.I18nCubeMessageExtractor):

    blacklist = devctl.I18nCubeMessageExtractor | {'static'}
    formats = devctl.I18nCubeMessageExtractor.formats + ['jinja2']

    def collect_jinja2(self):
        return self.find('.jinja2')

    def extract_jinja2(self, files):
        return self._xgettext(files, output='jinja.pot',
                              extraopts='-L python --from-code=utf-8')

Then, you’ll have to register it with a cubicweb.i18ncube entry point in your cube’s setup.py:

setup(
    # ...
    entry_points={
        # ...
        'cubicweb.i18ncube': [
            'mycube=cubicweb_mycube.mymodule:MyMessageExtractor',
        ],
    },
    # ...
)

Editing po files#

Using a PO aware editor#

Many tools exist to help maintain .po (PO) files. Common editors or development environment provides modes for these. One can also find dedicated PO files editor, such as poedit.

While usage of such a tool is commendable, PO files are perfectly editable with a (unicode aware) plain text editor. It is also useful to know their structure for troubleshooting purposes.

Structure of a PO file#

In this section, we selectively quote passages of the GNU gettext manual chapter on PO files, available there:

https://www.gnu.org/software/hello/manual/gettext/PO-Files.html

One PO file entry has the following schematic structure:

white-space
#  translator-comments
#. extracted-comments
#: reference...
#, flag...
#| msgid previous-untranslated-string
msgid untranslated-string
msgstr translated-string

A simple entry can look like this:

#: lib/error.c:116
msgid "Unknown system error"
msgstr "Error desconegut del sistema"

It is also possible to have entries with a context specifier. They look like this:

white-space
#  translator-comments
#. extracted-comments
#: reference...
#, flag...
#| msgctxt previous-context
#| msgid previous-untranslated-string
msgctxt context
msgid untranslated-string
msgstr translated-string

The context serves to disambiguate messages with the same untranslated-string. It is possible to have several entries with the same untranslated-string in a PO file, provided that they each have a different context. Note that an empty context string and an absent msgctxt line do not mean the same thing.

Contexts and CubicWeb#

CubicWeb PO files have both non-contextual and contextual msgids.

Contextual entries are automatically used in some cases. For instance, entity.dc_type(), eschema.display_name(req) or display_name(etype, req, form, context) methods/function calls will use them.

It is also possible to explicitly use a context with _cw.pgettext(context, msgid).

Specialize translation for an application cube#

Every cube has its own translation files. For a specific application cube it can be useful to specialize translations of other cubes. You can either mark those strings for translation using _ in the python code, or add a static-messages.pot file into the i18n directory. This file looks like:

msgid ""
msgstr ""
"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: pygettext.py 1.5\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

msgig "expression to be translated"
msgstr ""

Doing this, expression to be translated will be taken into account by the i18ncube command and additional messages will then appear in .po files of the cube.

The property mecanism#

Property API#

Registering and using your own property#

HTTP cache management#

Cache policies#

Exception#

Helper functions#

Locate resources#

Static files handling#

Pyramid#

cubicweb.pyramid provides a way to bind a CubicWeb data repository to a Pyramid WSGI web application.

It can be used in two different ways:

  • Through the pyramid command or through cubicweb.pyramid.wsgi_application() WSGI application factory, one can run an all-in-one CubicWeb instance with the web part served by a Pyramid application. This is referred to as the backwards compatible mode.

  • Through the pyramid configuration type, one can setup a CubicWeb instance which repository can be used from within a Pyramid application. Such an instance may be launched through pserve or any WSGI server as would any plain Pyramid application.

Quick start#

Prerequites#

Install the pyramid flavour of CubicWeb (here with pip, possibly in a virtualenv):

pip install cubicweb

Instance creation and running#

In backwards compatible mode#

In this mode, you can simply create an instance of kind all-in-one with the cubicweb-ctl create command. You’ll then need to add a pyramid.ini file in your instance directory, see Pyramid Settings file for details about the content of this file.

Start the instance with the ‘pyramid’ command instead of ‘start’:

cubicweb-ctl pyramid --debug myinstance

Without backwards compatibility#

In this mode, you can create an instance of kind pyramid as follow:

cubicweb-ctl create -c pyramid <cube_name> <instance_name>

This will bootstrap a development.ini file typical of a Pyramid application in the instance’s directory. The new instance may then be launched by any WSGI server, for instance with pserve:

pserve etc/cubicweb.d/<instance_name>/development.ini

In a pyramid application#

  • Create a pyramid application

  • Include cubicweb.pyramid:

    def includeme(config):
    # ...
    config.include('cubicweb.pyramid')
    # ...

  • Configure the instance name (in the .ini file):

    cubicweb.instance = myinstance

  • Configure the base-url in all-in-one.conf to match the ones of the pyramid configuration (this is a temporary limitation).

The ‘pyramid’ command#

The ‘pyramid’ command is a replacement for the ‘start’ command of cubicweb-ctl tool. It provides the same options and a few other ones.

Options#

--no-daemon#

Run the server in the foreground.

--debug-mode#

Activate the repository debug mode (logs in the console and the debug toolbar). Implies --no-daemon.

Also force the following pyramid options:

pyramid.debug_authorization = yes
pyramid.debug_notfound = yes
pyramid.debug_routematch = yes
pyramid.reload_templates = yes
-D, --debug#

Equals to --debug-mode --no-daemon --reload

--reload#

Restart the server if any source file is changed

--reload-interval=RELOAD_INTERVAL#

Interval, in seconds, between file modifications checks [current: 1]

-l <log level>, --loglevel=<log level>#

Set the loglevel. debug if -D is set, error otherwise

--profile-output=PROFILE_OUTPUT#

Profiling output file (default: “program.prof”)

--profile-dump-every=N#

Dump profile stats to ouput every N requests (default: 100)

Settings#

Cubicweb Settings#

Pyramid CubicWeb will not make use of the configuration entries found in the cubicweb configuration (a.k.a. all-in-one.conf) for any pyramid related configuration value.

Pyramid Settings file#

In backwards compatibility mode, Pyramid settings will be looked for in a pyramid.ini file in the instance home directory (where the all-in-one.conf file is), its [main] section will be read and used as the settings of the pyramid Configurator.

This configuration file is almost the same as the one read by pserve, which allow to easily add any pyramid extension and configure it.

A typical pyramid.ini file is:

[main]
pyramid.includes =
    pyramid_redis_sessions

cubicweb.defaults = no
cubicweb.includes =
    cubicweb.pyramid.auth
    cubicweb.pyramid.login

cubicweb.profile = no

redis.sessions.secret = your_cookie_signing_secret
redis.sessions.timeout = 1200

redis.sessions.host = mywheezy

Without backwards compatibility a standard development.ini file can be used with any useful CubicWeb-specific settings added.

Pyramid CubicWeb configuration entries#

The Pyramid CubicWeb specific configuration entries are:

cubicweb.instance (string)#

A CubicWeb instance name. Useful when the application is not run by The ‘pyramid’ command.

cubicweb.debug (bool)#

Enables the cubicweb debugmode. Works only if the instance is setup by cubicweb.instance.

Unlike when the debugmode is set by the cubicweb-ctl pyramid --debug-mode command, the pyramid debug options are untouched.

cubicweb.includes (list)#

Same as pyramid.includes, but the includes are done after the cubicweb specific registry entries are initialized.

Useful to include extensions that requires these entries.

cubicweb.bwcompat (bool)#

(True) Enable/disable backward compatibility. This only applies to “all-in-one” configuration type.

See cubicweb.pyramid.bwcompat.

cubicweb.bwcompat.errorhandler (bool)#

(True) Enable/disable the backward compatibility error handler. Set to ‘no’ if you need to define your own error handlers.

cubicweb.defaults (bool)#

(True) Enable/disable defaults. See cubicweb.pyramid.defaults.

cubicweb.auth.update_login_time (bool)#

(True) Add a cubicweb.pyramid.auth.UpdateLoginTimeAuthenticationPolicy policy, that update the CWUser.login_time attribute when a user login.

cubicweb.auth.authtkt (bool)#

(True) Enables the 2 cookie-base auth policies, which activate/deactivate depending on the persistent argument passed to remember.

The default login views set persistent to True if a __setauthcookie parameters is passed to them, and evals to True in pyramid.settings.asbool().

The configuration values of the policies are arguments for pyramid.authentication.AuthTktAuthenticationPolicy.

The first policy handles session authentication. It doesn’t get activated if remember() is called with persistent=False:

cubicweb.auth.authtkt.session.cookie_name (str)#

(‘auth_tkt’) The cookie name. Must be different from the persistent authentication cookie name.

cubicweb.auth.authtkt.session.samesite (str)#

(‘auth_tkt’) Allows you to declare if your cookie should be restricted to a first-party or same-site context. See here for more information.

cubicweb.auth.authtkt.session.timeout (int)#

  1. Cookie timeout.

cubicweb.auth.authtkt.session.reissue_time (int)#

  1. Reissue time.

The second policy handles persistent authentication. It doesn’t get activated if remember() is called with persistent=True:

cubicweb.auth.authtkt.persistent.cookie_name (str)#

(‘auth_tkt’) The cookie name. Must be different from the session authentication cookie name.

cubicweb.auth.authtkt.persistent.samesite (str)#

(‘auth_tkt’) Allows you to declare if your cookie should be restricted to a first-party or same-site context. See here for more information.

cubicweb.auth.authtkt.persistent.max_age (int)#

(30 days) Max age in seconds.

cubicweb.auth.authtkt.persistent.reissue_time (int)#

(1 day) Reissue time in seconds.

Both policies set the secure flag to True by default, meaning that cookies will only be sent back over a secure connection (see Authentication Policies documentation for details). This can be configured through cubicweb.auth.authtkt.persistent.secure and cubicweb.auth.authtkt.session.secure configuration options.

cubicweb.auth.groups_principals (bool)#

(True) Setup a callback on the authentication stack that inject the user groups in the principals.

Authentication#

Overview#

A default authentication stack is provided by the cubicweb.pyramid.auth module, which is included by cubicweb.pyramid.default.

The authentication stack is built around pyramid_multiauth, and provides a few default policies that reproduce the default cubicweb behavior.

Note

Note that this module only provides an authentication policy, not the views that handle the login form. See cubicweb.pyramid.login

Customize#

The default policies can be individually deactivated, as well as the default authentication callback that returns the current user groups as principals.

The following settings can be set to False:

  • cubicweb.auth.update_login_time. Activate the policy that update the user login_time when remember is called.

  • cubicweb.auth.authtkt and all its subvalues.

  • cubicweb.auth.groups_principals

Additionnal policies can be added by accessing the MultiAuthenticationPolicy instance in the registry:

mypolicy = SomePolicy()
authpolicy = config.registry['cubicweb.authpolicy']
authpolicy._policies.append(mypolicy)

The pyramid debug toolbar#

The pyramid webserver comes with an integrated debug toolbar that offers a lot of information to ease development. To ease the development process in CubicWeb a series of custom debug panels have been developped especially for that purpose.

To use the pyramid debug toolbar in CubicWeb, you need to:

  • install it either by doing a pip install pyramid_debugtoolbar or following the official installation instructions

  • launch the pyramid command adding the -t/–toolbar argument to enable it like so: cubicweb-ctl pyramid my_instance -t (you probably want to add -D to activate the debug mode during development)

  • navigate to the website and click on the icon on the right like on this screenshot:

images/debug_toolbar_icon.png

And you’ll have access to the debug toolbar content for this page.

Custom panels#

A series of custom debug panels have been written to offer more useful debug information during development. Here is the list:

General ‘CubicWeb’ Panel#

Provides:

  • currently selected controller for this with and uri/requests information

  • current instance configuration, options that differs from default ones are in bold

  • a list of useful links like on the default CW home

Screenshot:

_images/debugtoolbar_general_panel.png

Registry Decisions Panel#

Provides:

  • a list of all decisions taken in all registry during this page construction

  • the arguments given to take the decision

  • all the selection entities during decisions with their score

  • which one has won if any

_images/debugtoolbar_registry_decisions_panel.png

Registry Store#

Provides:

  • a listing of all the content of the different registries

  • for each entity its detailed information

_images/debugtoolbar_registry_content_panel.png

RQL#

Provides:

  • a list of all executed RQL queries during this page creation

  • for each RQL query all the generated SQL queries

  • detail information like the result, the args and the description of each query

  • the call stack on each query to see where it has been called

_images/debugtoolbar_rql_panel.png _images/debugtoolbar_rql_traceback_panel.png

SQL#

Provides:

  • a list of all executed SQL queries during this page creation

  • for each SQL query the RQL query that has generated it, if any (some aren’t)

  • detail information like the result, the args and if the query has rollback

  • the call stack on each query to see where it has been called

_images/debugtoolbar_sql_panel.png

Accessing the sources of the class/functions/method listing the debug panels#

A traversal of all those custom panels is the see the source code of all listing class/functions/methods. You can access those by:

  • clicking on the [source] close to the target when available

  • clicking on the file path in the traceback stack

_images/debugtoolbar_show_source_link.png _images/debugtoolbar_traceback_source_link.png

You be sent to a page looking like this:

_images/debugtoolbar_show_source.png

Contributing#

All source code for the custom panels is located here and the documentation of how to write custom toolbar panels here.

Additional Services#

In this chapter, we introduce services crossing the web - repository - administration organisation of the first parts of the CubicWeb book. Those services can be either proper services (like the undo functionality) or mere topical cross-sections across CubicWeb.

Undoing changes in CubicWeb#

Many desktop applications offer the possibility for the user to undo its last changes : this undo feature has now been integrated into the CubicWeb framework. This document will introduce you to the undo feature both from the end-user and the application developer point of view.

But because a semantic web application and a common desktop application are not the same thing at all, especially as far as undoing is concerned, we will first introduce what is the undo feature for now.

What’s undoing in a CubicWeb application#

What is an undo feature is quite intuitive in the context of a desktop application. But it is a bit subtler in the context of a Semantic Web application. This section introduces some of the main differences between a classical desktop and a Semantic Web applications to keep in mind in order to state precisely what we want.

The notion transactions#

A CubicWeb application acts upon an Entity-Relationship model, described by a schema. This allows to ensure some data integrity properties. It also implies that changes are made by all-or-none groups called transactions, such that the data integrity is preserved whether the transaction is completely applied or none of it is applied.

A transaction can thus include more actions than just those directly required by the main purpose of the user. For example, when a user just writes a new blog entry, the underlying transaction holds several actions as illustrated below :

  • By admin on 2012/02/17 15:18 - Created Blog entry : Torototo

    1. Created Blog entry : Torototo

    2. Added relation : Torototo owned by admin

    3. Added relation : Torototo blog entry of Undo Blog

    4. Added relation : Torototo in state draft (draft)

    5. Added relation : Torototo created by admin

Because of the very nature (all-or-none) of the transactions, the “undoable stuff” are the transactions and not the actions !

Public and private actions within a transaction#

Actually, within the transaction “Created Blog entry : Torototo”, two of those actions are said to be public and the others are said to be private. Public here means that the public actions (1 and 3) were directly requested by the end user ; whereas private means that the other actions (2, 4, 5) were triggered “under the hood” to fulfill various requirements for the user operation (ensuring integrity, security, … ).

And because quite a lot of actions can be triggered by a “simple” end-user request, most of which the end-user is not (and does not need or wish to be) aware, only the so-called public actions will appear 1 in the description of the an undoable transaction.

  • By admin on 2012/02/17 15:18 - Created Blog entry : Torototo

    1. Created Blog entry : Torototo

    2. Added relation : Torototo blog entry of Undo Blog

But note that both public and private actions will be undone together when the transaction is undone.

(In)dependent transactions : the simple case#

A CubicWeb application can be used simultaneously by different users (whereas a single user works on an given office document at a given time), so that there is not always a single history time-line in the CubicWeb case. Moreover CubicWeb provides security through the mechanism of permissions granted to each user. This can lead to some transactions not being undoable in some contexts.

In the simple case two (unprivileged) users Alice and Bob make relatively independent changes : then both Alice and Bob can undo their changes. But in some case there is a clean dependency between Alice’s and Bob’s actions or between actions of one of them. For example let’s suppose that :

  • Alice has created a blog,

  • then has published a first post inside,

  • then Bob has published a second post in the same blog,

  • and finally Alice has updated its post contents.

Then it is clear that Alice can undo her contents changes and Bob can undo his post creation independently. But Alice can not undo her post creation while she has not first undone her changes. It is also clear that Bob should not have the permissions to undo any of Alice’s transactions.

More complex dependencies between transactions#

But more surprising things can quickly happen. Going back to the previous example, Alice can undo the creation of the blog after Bob has published its post in it ! But this is possible only because the schema does not require for a post to be in a blog. Would the blog entry of relation have been mandatory, then Alice could not have undone the blog creation because it would have broken integrity constraint for Bob’s post.

When a user attempts to undo a transaction the system will check whether a later transaction has explicit dependency on the would-be-undone transaction. In this case the system will not even attempt the undo operation and inform the user.

If no such dependency is detected the system will attempt the undo operation but it can fail, typically because of integrity constraint violations. In such a case the undo operation is completely 3 rollbacked.

The undo feature for CubicWeb end-users#

The exposition of the undo feature to the end-user through a Web interface is still quite basic and will be improved toward a greater usability. But it is already fully functional. For now there are two ways to access the undo feature as long as the it has been activated in the instance configuration file with the option undo-support=yes.

Immediately after having done the change to be canceled through the undo link in the message. This allows to undo an hastily action immediately. For example, just after having validated the creation of the blog entry A second blog entry we get the following message, allowing to undo the creation.

Screenshot of the undo link in the message

At any time we can access the undo-history view accessible from the start-up page.

Screenshot of the startup menu with access to the history view

This view will provide inspection of the transaction and their (public) actions. Each transaction provides its own undo link. Only the transactions the user has permissions to see and undo will be shown.

Screenshot of the undo history main view

If the user attempts to undo a transaction which can’t be undone or whose undoing fails, then a message will explain the situation and no partial undoing will be left behind.

This is all for the end-user side of the undo mechanism : this is quite simple indeed ! Now, in the following section, we are going to introduce the developer side of the undo mechanism.

The undo feature for CubicWeb application developers#

A word of warning : this section is intended for developers, already having some knowledge of what’s under CubicWeb’s hood. If it is not yet the case, please refer to CubicWeb documentation http://docs.cubicweb.org/ .

Overview#

The core of the undo mechanisms is at work in the native source, beyond the RQL. This does mean that transactions and actions are no entities. Instead they are represented at the SQL level and exposed through the DB-API supported by the repository Connection objects.

Once the undo feature has been activated in the instance configuration file with the option undo-support=yes, each mutating operation (cf. 2) will be recorded in some special SQL table along with its associated transaction. Transaction are identified by a txuuid through which the functions of the DB-API handle them.

On the web side the last commited transaction txuuid is remembered in the request’s data to allow for imediate undoing whereas the undo-history view relies upon the DB-API to list the accessible transactions. The actual undoing is performed by the UndoController accessible at URL of the form www.my.host/my/instance/undo?txuuid=…

The repository side#

Please refer to the file cubicweb/server/sources/native.py and cubicweb/transaction.py for the details.

The undoing information is mainly stored in three SQL tables:

transactions

Stores the txuuid, the user eid and the date-and-time of the transaction. This table is referenced by the two others.

tx_entity_actions

Stores the undo information for actions on entities.

tx_relation_actions

Stores the undo information for the actions on relations.

When the undo support is activated, entries are added to those tables for each mutating operation on the data repository, and are deleted on each transaction undoing.

Those table are accessible through the following methods of the repository Connection object :

undoable_transactions

Returns a list of Transaction objects accessible to the user and according to the specified filter(s) if any.

tx_info

Returns a Transaction object from a txuuid

undo_transaction

Returns the list of Action object for the given txuuid.

NB: By default it only return public actions.

The web side#

The exposure of the undo feature to the end-user through the Web interface relies on the DB-API introduced above. This implies that the transactions and actions are not entities linked by relations on which the usual views can be applied directly.

That’s why the file cubicweb/web/views/undohistory.py defines some dedicated views to access the undo information :

UndoHistoryView

This is a StartupView, the one accessible from the home page of the instance which list all transactions.

UndoableTransactionView

This view handles the display of a single Transaction object.

UndoableActionBaseView

This (abstract) base class provides private methods to build the display of actions whatever their nature.

Undoable[Add|Remove|Create|Delete|Update]ActionView

Those views all inherit from UndoableActionBaseView and each handles a specific kind of action.

UndoableActionPredicate

This predicate is used as a selector to pick the appropriate view for actions.

Apart from this main undo-history view a txuuid is stored in the request’s data last_undoable_transaction in order to allow immediate undoing of a hastily validated operation. This is handled in cubicweb/web/application.py in the main_publish and add_undo_link_to_msg methods for the storing and displaying respectively.

Once the undo information is accessible, typically through a txuuid in an undo URL, the actual undo operation can be performed by the UndoController defined in cubicweb/web/views/basecontrollers.py. This controller basically extracts the txuuid and performs a call to undo_transaction and in case of an undo-specific error, lets the top level publisher handle it as a validation error.

Conclusion#

The undo mechanism relies upon a low level recording of the mutating operation on the repository. Those records are accessible through some method added to the DB-API and exposed to the end-user either through a whole history view of through an immediate undoing link in the message box.

The undo feature is functional but the interface and configuration options are still quite reduced. One major improvement would be to be able to filter with a finer grain which transactions or actions one wants to see in the undo-history view. Another critical improvement would be to enable the undo feature on a part only of the entity-relationship schema to avoid storing too much useless data and reduce the underlying overhead.

But both functionality are related to the strong design choice not to represent transactions and actions as entities and relations. This has huge benefits in terms of safety and conceptual simplicity but prevents from using lots of convenient CubicWeb features such as facets to access undo information.

Before developing further the undo feature or eventually revising this design choice, it appears that some return of experience is strongly needed. So don’t hesitate to try the undo feature in your application and send us some feedback.

Notes#

1

The end-user Web interface could be improved to enable user to choose whether he wishes to see private actions.

2

There is only five kind of elementary actions (beyond merely accessing data for reading):

  • C : creating an entity

  • D : deleting an entity

  • U : updating an entity attributes

  • A : adding a relation

  • R : removing a relation

3

Meaning none of the actions in the transaction is undone. Depending upon the application, it might make sense to enable partial undo. That is to say undo in which some actions could not be undo without preventing to undo the others actions in the transaction (as long as it does not break schema integrity). This is not forbidden by the back-end but is deliberately not supported by the front-end (for now at least).

Appendixes#

The following chapters are reference material.

Frequently Asked Questions (FAQ)#

Generalities#

Why do you use the LGPL license to prevent me from doing X ?#

LGPL means that if you redistribute your application, you need to redistribute the changes you made to CubicWeb under the LGPL licence.

Publishing a web site has nothing to do with redistributing source code according to the terms of the LGPL. A fair amount of companies use modified LGPL code for internal use. And someone could publish a CubicWeb component under a BSD licence for others to plug into a LGPL framework without any problem. The only thing we are trying to prevent here is someone taking the framework and packaging it as closed source to his own clients.

Why does not CubicWeb have a template language ?#

There are enough template languages out there. You can use your preferred template language if you want.

CubicWeb does not define its own templating language as this was not our goal. Based on our experience, we realized that we could gain productivity by letting designers use design tools and developpers develop without the use of the templating language as an intermediary that could not be anyway efficient for both parties. Python is the templating language that we use in CubicWeb, but again, it does not prevent you from using a templating language.

Moreover, CubicWeb currently supports simpletal out of the box and it is also possible to use the cwtags library to build html trees using the with statement with more comfort than raw strings.

Why do you think using pure python is better than using a template language ?#

Python is an Object Oriented Programming language and as such it already provides a consistent and strong architecture and syntax a templating language would not reach.

Using Python instead of a template langage for describing the user interface makes it to maintain with real functions/classes/contexts without the need of learning a new dialect. By using Python, we use standard OOP techniques and this is a key factor in a robust application.

CubicWeb looks pretty recent. Is it stable ?#

It is constantly evolving, piece by piece. The framework has evolved since 2001 and data has been migrated from one schema to the other ever since. There is a well-defined way to handle data and schema migration.

You can see the roadmap there: https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/boards.

Why is the RQL query language looking similar to X ?#

It may remind you of SQL but it is higher level than SQL, more like SPARQL. Except that SPARQL did not exist when we started the project. With version 3.4, CubicWeb has support for SPARQL.

The RQL language is what is going to make a difference with django- like frameworks for several reasons.

  1. accessing data is much easier with it. One can write complex queries with RQL that would be tedious to define and hard to maintain using an object/filter suite of method calls.

  2. it offers an abstraction layer allowing your applications to run on multiple back-ends. That means not only various SQL backends (postgresql, sqlite, sqlserver, mysql), but also non-SQL data stores like LDAP directories and subversion/mercurial repositories (see the vcsfile component).

Which ajax library is CubicWeb using ?#

CubicWeb uses jQuery and provides a few helpers on top of that. Additionally, some jQuery plugins are provided (some are provided in specific cubes).

Development#

How to create an anonymous user ?#

This allows to browse the site without being authenticated. In the all-in-one.conf file of your instance, define the anonymous user as follows

# login of the CubicWeb user account to use for anonymous user (if you want to
# allow anonymous)
anonymous-user=anon

# password of the CubicWeb user account matching login
anonymous-password=anon

You also must ensure that this anon user is a registered user of the DB backend. If not, you can create through the administation interface of your instance by adding a user with in the group guests.

Note

While creating a new instance, you can decide to allow access to anonymous user, which will automatically execute what is decribed above.

How to format an entity date attribute ?#

If your schema has an attribute of type Date or Datetime, you usually want to format it when displaying it. First, you should define your preferred format using the site configuration panel http://appurl/view?vid=systempropertiesform and then set ui.date and/or ui.datetime. Then in the view code, use:

entity.printable_value(date_attribute)

which will always return a string whatever the attribute’s type (so it’s recommended also for other attribute types). By default it expects to generate HTML, so it deals with rich text formating, xml escaping…

How to update a database after a schema modification ?#

It depends on what has been modified in the schema.

  • update the permissions and properties of an entity or a relation: sync_schema_props_perms('MyEntityOrRelation').

  • add an attribute: add_attribute('MyEntityType', 'myattr').

  • add a relation: add_relation_definition('SubjRelation', 'MyRelation', 'ObjRelation').

I get NoSelectableObject exceptions, how do I debug selectors ?#

You just need to put the appropriate context manager around view/component selection. One standard place for components is in cubicweb/vregistry.py:

def possible_objects(self, *args, **kwargs):
    """return an iterator on possible objects in this registry for the given
    context
    """
    from logilab.common.registry import traced_selection
    with traced_selection():
        for appobjects in self.itervalues():
            try:
                yield self._select_best(appobjects, *args, **kwargs)
            except NoSelectableObject:
                continue

This will yield additional WARNINGs, like this:

2009-01-09 16:43:52 - (cubicweb.selectors) WARNING: selector one_line_rset returned 0 for <class 'cubicweb.web.views.basecomponents.WFHistoryVComponent'>

For views, you can put this context in cubicweb/web/views/basecontrollers.py in the ViewController:

def _select_view_and_rset(self, rset):
    ...
    try:
        from logilab.common.registry import traced_selection
        with traced_selection():
            view = self._cw.vreg['views'].select(vid, req, rset=rset)
    except ObjectNotFound:
        self.warning("the view %s could not be found", vid)
        req.set_message(req._("The view %s could not be found") % vid)
        vid = vid_from_rset(req, rset, self._cw.vreg.schema)
        view = self._cw.vreg['views'].select(vid, req, rset=rset)
    ...

I get “database is locked” when executing tests#

If you have “database is locked” as error when you are executing security tests, it is usually because commit or rollback are missing before login() calls.

You can also use a context manager, to avoid such errors, as described here: Managing connections or users.

What are hooks used for ?#

Hooks are executed around (actually before or after) events. The most common events are data creation, update and deletion. They permit additional constraint checking (those not expressible at the schema level), pre and post computations depending on data movements.

As such, they are a vital part of the framework.

Other kinds of hooks, called Operations, are available for execution just before commit.

For more information, read Hooks section.

Configuration#

How to configure a LDAP source ?#

See LDAP integration.

How to import LDAP users in CubicWeb ?#

Here is a useful script which enables you to import LDAP users into your CubicWeb instance by running the following:

import os
import pwd
import sys

from logilab.database import get_connection

def getlogin():
    """avoid using os.getlogin() because of strange tty/stdin problems
    (man 3 getlogin)
    Another solution would be to use $LOGNAME, $USER or $USERNAME
    """
    return pwd.getpwuid(os.getuid())[0]


try:
    database = sys.argv[1]
except IndexError:
    print 'USAGE: python ldap2system.py <database>'
    sys.exit(1)

if input('update %s db ? [y/n]: ' % database).strip().lower().startswith('y'):
    cnx = get_connection(user=getlogin(), database=database)
    cursor = cnx.cursor()

    insert = ('INSERT INTO euser (creation_date, eid, modification_date, login, '
              ' firstname, surname, last_login_time, upassword) '
              "VALUES (%(mtime)s, %(eid)s, %(mtime)s, %(login)s, %(firstname)s, "
              "%(surname)s, %(mtime)s, './fqEz5LeZnT6');")
    update = "UPDATE entities SET source='system' WHERE eid=%(eid)s;"
    cursor.execute("SELECT eid,type,source,extid,mtime FROM entities WHERE source!='system'")
    for eid, type, source, extid, mtime in cursor.fetchall():
        if type != 'CWUser':
            print "don't know what to do with entity type", type
            continue
        if source != 'ldapuser':
            print "don't know what to do with source type", source
            continue
        ldapinfos = dict(x.strip().split('=') for x in extid.split(','))
        login = ldapinfos['uid']
        firstname = ldapinfos['uid'][0].upper()
        surname = ldapinfos['uid'][1:].capitalize()
        if login != 'jcuissinat':
            args = dict(eid=eid, type=type, source=source, login=login,
                        firstname=firstname, surname=surname, mtime=mtime)
            print args
            cursor.execute(insert, args)
            cursor.execute(update, args)

    cnx.commit()
    cnx.close()

Security#

How to reset the password for user joe ?#

If you want to reset the admin password for myinstance, do:

$ cubicweb-ctl reset-admin-pwd myinstance

You need to generate a new encrypted password:

$ python
>>> from cubicweb.server.utils import crypt_password
>>> crypt_password('joepass')
'qHO8282QN5Utg'
>>>

and paste it in the database:

$ psql mydb
mydb=> update cw_cwuser set cw_upassword='qHO8282QN5Utg' where cw_login='joe';
UPDATE 1

if you’re running over SQL Server, you need to use the CONVERT function to convert the string to varbinary(255). The SQL query is therefore:

update cw_cwuser set cw_upassword=CONVERT(varbinary(255), 'qHO8282QN5Utg') where cw_login='joe';

Be careful, the encryption algorithm is different on Windows and on Unix. You cannot therefore use a hash generated on Unix to fill in a Windows database, nor the other way round.

You can prefer use a migration script similar to this shell invocation instead:

$ cubicweb-ctl shell <instance>
>>> from cubicweb import Binary
>>> from cubicweb.server.utils import crypt_password
>>> crypted = crypt_password('joepass')
>>> rset = rql('Any U WHERE U is CWUser, U login "joe"')
>>> joe = rset.get_entity(0,0)
>>> joe.cw_set(upassword=Binary(crypted))

Please, refer to the script example is provided in the misc/examples/chpasswd.py file.

The more experimented people would use RQL request directly:

>>> rql('SET X upassword %(a)s WHERE X is CWUser, X login "joe"',
...     {'a': crypted})

I’ve just created a user in a group and it doesn’t work !#

You are probably getting errors such as

remove {'PR': 'Project', 'C': 'CWUser'} from solutions since your_user has no read access to cost

This is because you have to put your user in the “users” group. The user has to be in both groups.

How is security implemented ?#

The basis for security is a mapping from operations to groups or arbitrary RQL expressions. These mappings are scoped to entities and relations.

This is an example for an Entity Type definition:

class Version(EntityType):
    """a version is defining the content of a particular project's
    release"""
    # definition of attributes is voluntarily missing
    __permissions__ = {'read': ('managers', 'users', 'guests',),
                       'update': ('managers', 'logilab', 'owners'),
                       'delete': ('managers',),
                       'add': ('managers', 'logilab',
                               ERQLExpression('X version_of PROJ, U in_group G, '
                                              'PROJ require_permission P, '
                                              'P name "add_version", P require_group G'),)}

The above means that permission to read a Version is granted to any user that is part of one of the groups ‘managers’, ‘users’, ‘guests’. The ‘add’ permission is granted to users in group ‘managers’ or ‘logilab’ or to users in group G, if G is linked by a permission entity named “add_version” to the version’s project.

An example for a Relation Definition (RelationType both defines a relation type and implicitly one relation definition, on which the permissions actually apply):

class version_of(RelationType):
    """link a version to its project. A version is necessarily linked
    to one and only one project. """
    # some lines voluntarily missing
    __permissions__ = {'read': ('managers', 'users', 'guests',),
                       'delete': ('managers', ),
                       'add': ('managers', 'logilab',
                               RRQLExpression('O require_permission P, P name "add_version", '
                                              'U in_group G, P require_group G'),) }

The main difference lies in the basic available operations (there is no ‘update’ operation) and the usage of an RRQLExpression (rql expression for a relation) instead of an ERQLExpression (rql expression for an entity).

You can find additional information in the section The security model.

Is it possible to bypass security from the UI (web front) part ?#

No. Only Hooks/Operations can do that.

Can PostgreSQL and CubicWeb authentication work with kerberos ?#

If you have PostgreSQL set up to accept kerberos authentication, you can set the db-host, db-name and db-user parameters in the sources configuration file while leaving the password blank. It should be enough for your instance to connect to postgresql with a kerberos ticket.

Relation Query Language (RQL)#

This chapter describes the Relation Query Language syntax and its implementation in CubicWeb.

Introduction#

Goals of RQL#

The goal is to have a semantic language in order to:

  • query relations in a clear syntax

  • empowers access to data repository manipulation

  • making attributes/relations browsing easy

As such, attributes will be regarded as cases of special relations (in terms of usage, the user should see no syntactic difference between an attribute and a relation).

Comparison with existing languages#

SQL#

RQL may remind of SQL but works at a higher abstraction level (the CubicWeb framework generates SQL from RQL to fetch data from relation databases). RQL is focused on browsing relations. The user needs only to know about the CubicWeb data model he is querying, but not about the underlying SQL model.

Sparql#

The query language most similar to RQL is SPARQL, defined by the W3C to serve for the semantic web.

Versa#

We should look in more detail, but here are already some ideas for the moment … Versa is the language most similar to what we wanted to do, but the model underlying data being RDF, there are some things such as namespaces or handling of the RDF types which does not interest us. On the functionality level, Versa is very comprehensive including through many functions of conversion and basic types manipulation, which we may want to look at one time or another. Finally, the syntax is a little esoteric.

Datalog#

Datalog is a prolog derived query langage which applies to relational databases. It is more expressive than RQL in that it accepts either extensional and intensional predicates (or relations). As of now, RQL only deals with intensional relations.

The different types of queries#

Search (Any)

Extract entities and attributes of entities.

Insert entities (INSERT)

Insert new entities or relations in the database. It can also directly create relationships for the newly created entities.

Update entities, create relations (SET)

Update existing entities in the database, or create relations between existing entities.

Delete entities or relationship (DELETE)

Remove entities or relations existing in the database.

RQL relation expressions#

RQL expressions apply to a live database defined by a Yams schema. Apart from the main type, or head, of the expression (search, insert, etc.) the most common constituent of an RQL expression is a (set of) relation expression(s).

An RQL relation expression contains three components:

  • the subject, which is an entity type

  • the predicate, which is a relation definition (an arc of the schema)

  • the object, which is either an attribute or a relation to another entity

<subject> <predicate> <object>

Warning

A relation is always expressed in the order: subject, predicate, object.

It is important to determine if the entity type is subject or object to construct a valid expression. Inverting the subject/object is an error since the relation cannot be found in the schema.

If one does not have access to the code, one can find the order by looking at the schema image in manager views (the subject is located at the beginning of the arrow).

An example of two related relation expressions:

P works_for C, P name N

RQL variables represent typed entities. The type of entities is either automatically inferred (by looking at the possible relation definitions, see Relation definition) or explicitely constrained using the is meta relation.

In the example above, we barely need to look at the schema. If variable names (in the RQL expression) and relation type names (in the schema) are expresssively designed, the human reader can infer as much as the CubicWeb querier.

The P variable is used twice but it always represent the same set of entities. Hence P works_for C and P name N must be compatible in the sense that all the Ps (which can refer to different entity types) must accept the works_for and name relation types. This does restrict the set of possible values of P.

Adding another relation expression:

P works_for C, P name N, C name "logilab"

This further restricts the possible values of P through an indirect constraint on the possible values of C. The RQL-level unification happening there is translated to one (or several) joins at the database level.

Note

In CubicWeb, the term relation is often found without ambiguity instead of predicate. This predicate is also known as the property of the triple in RDF concepts

RQL Operators#

An RQL expression’s head can be completed using various operators such as ORDERBY, GROUPBY, HAVING, LIMIT etc.

RQL relation expressions can be grouped with UNION or WITH. Predicate oriented keywords such as EXISTS, OR, NOT are available.

The complete zoo of RQL operators is described extensively in the following chapter (RQL syntax).

RQL syntax#

Reserved keywords#

AND, ASC, BEING, DELETE, DESC, DISTINCT, EXISTS, FALSE, GROUPBY,
HAVING, ILIKE, INSERT, LIKE, LIMIT, NOT, NOW, NULL, NULLSFIRST, NULLSLAST,
OFFSET, OR, ORDERBY, SET, TODAY, TRUE, UNION, WHERE, WITH

The keywords are not case sensitive. You should not use them when defining your schema, or as RQL variable names.

Case#

  • Variables should be all upper-cased.

  • Relation should be all lower-cased and match exactly names of relations defined in the schema.

  • Entity types should start with an upper cased letter and be followed by at least a lower cased latter.

Variables and typing#

Entities and values to browse and/or select are represented in the query by variables that must be written in capital letters.

With RQL, we do not distinguish between entities and attributes. The value of an attribute is considered as an entity of a particular type (see below), linked to one (real) entity by a relation called the name of the attribute, where the entity is the subject and the attribute the object.

The possible type(s) for each variable is derived from the schema according to the constraints expressed above and thanks to the relations between each variable.

We can restrict the possible types for a variable using the special relation is in the restrictions.

Virtual relations#

Those relations may only be used in RQL query but are not actual attributes of your entities.

  • has_text: relation to use to query the full text index (only for entities having fulltextindexed attributes).

  • identity: relation to use to tell that a RQL variable is the same as another when you’ve to use two different variables for querying purpose. On the opposite it’s also useful together with the NOT operator to tell that two variables should not identify the same entity

Literal expressions#

Bases types supported by RQL are those supported by yams schema. Literal values are expressed as explained below:

  • string should be between double or single quotes. If the value contains a quote, it should be preceded by a backslash ‘\’

  • floats separator is dot ‘.’

  • boolean values are TRUE and FALSE keywords

  • date and time should be expressed as a string with ISO notation : YYYY/MM/DD [hh:mm], or using keywords TODAY and NOW

You may also use the NULL keyword, meaning ‘unspecified’.

Operators#

Logical operators#
AND, OR, NOT, ','

‘,’ is equivalent to ‘AND’ but with the smallest among the priority of logical operators (see Operators priority).

Mathematical operators#

Operator

Description

Example

Result

+

addition

2 + 3

5

-

subtraction

2 - 3

-1

*

multiplication

2 * 3

6

/

division

4 / 2

2

%

modulo (remainder)

5 % 4

1

^

exponentiation

2.0 ^ 3.0

8

&

bitwise AND

91 & 15

11

|

bitwise OR

32 | 3

35

#

bitwise XOR

17 # 5

20

~

bitwise NOT

~1

-2

<<

bitwise shift left

1 << 4

16

>>

bitwise shift right

8 >> 2

2

Notice integer division truncates results depending on the backend behaviour. For instance, postgresql does.

Comparison operators#
=, !=, <, <=, >=, >, IN

The syntax to use comparison operators is:

VARIABLE attribute <operator> VALUE

The = operator is the default operator and can be omitted, i.e. :

VARIABLE attribute = VALUE

is equivalent to

VARIABLE attribute VALUE

The operator IN provides a list of possible values:

Any X WHERE X name IN ('chauvat', 'fayolle', 'di mascio', 'thenault')
String operators#
LIKE, ILIKE, ~=, REGEXP

The LIKE string operator can be used with the special character % in a string as wild-card:

-- match every entity whose name starts with 'Th'
Any X WHERE X name ~= 'Th%'
-- match every entity whose name endswith 'lt'
Any X WHERE X name LIKE '%lt'
-- match every entity whose name contains a 'l' and a 't'
Any X WHERE X name LIKE '%l%t%'

ILIKE is the case insensitive version of LIKE. It’s not available on all backend (e.g. sqlite doesn’t support it). If not available for your backend, ILIKE will behave like LIKE.

~= is a shortcut version of ILIKE, or of LIKE when the former is not available on the back-end.

The REGEXP is an alternative to LIKE that supports POSIX regular expressions:

-- match entities whose title starts with a digit
Any X WHERE X title REGEXP "^[0-9].*"

The underlying SQL operator used is back-end-dependent :

  • the ~ operator is used for postgresql,

  • the REGEXP operator for mysql and sqlite.

Other back-ends are not supported yet.

Operators priority#

  1. (, )

  2. ^, <<, >>

  3. *, /, %, &

  4. +, -, |, #

  5. NOT

  6. AND

  7. OR

  8. ,

Search Query#

Simplified grammar of search query:

[ `DISTINCT`] `Any` V1 (, V2) \*
[ `GROUPBY` V1 (, V2) \*] [ `ORDERBY` <orderterms>]
[ `LIMIT` <value>] [ `OFFSET` <value>]
[ `WHERE` <triplet restrictions>]
[ `WITH` V1 (, V2)\* BEING (<query>)]
[ `HAVING` <other restrictions>]
[ `UNION` <query>]
Selection#

The fist occuring clause is the selection of terms that should be in the result set. Terms may be variable, literals, function calls, arithmetic, etc. and each term is separated by a comma.

There will be as much column in the result set as term in this clause, respecting order.

Syntax for function call is somewhat intuitive, for instance:

Any UPPER(N) WHERE P firstname N
Grouping and aggregating#

The GROUPBY keyword is followed by a list of terms on which results should be grouped. They are usually used with aggregate functions, responsible to aggregate values for each group (see Aggregate functions).

For grouped queries, all selected variables must be either aggregated (i.e. used by an aggregate function) or grouped (i.e. listed in the GROUPBY clause).

Sorting#

The ORDERBY keyword if followed by the definition of the selection order: variable or column number followed by sorting method (ASC, DESC), ASC being the default. If the sorting method is not specified, then the sorting is ascendant (ASC).

It is also possible to precise a specific ordering for NULL values. The NULLSFIRST and NULLSLAST options can be used to determine whether nulls appear before or after non-null values in the sort ordering. By default, null values sort as if larger than any non-null value; that is, NULLSFIRST is the default for DESC order, and NULLSLAST otherwise. These options are written after the sorting method when it is specified. For instance, this request will return all projects ordered by creation date in descending order, with projects with no date in last position.

Any X ORDERBY Y DESC NULLSLAST WHERE X creation_date Y
Pagination#

The LIMIT and OFFSET keywords may be respectively used to limit the number of results and to tell from which result line to start (for instance, use LIMIT 20 to get the first 20 results, then LIMIT 20 OFFSET 20 to get the next 20.

Restrictions#

The WHERE keyword introduce one of the “main” part of the query, where you “define” variables and add some restrictions telling what you’re interested in.

It’s a list of triplets “subject relation object”, e.g. V1 relation (V2 | <static value>). Triplets are separated using Logical operators.

Note

About the negation operator (NOT):

  • NOT X relation Y is equivalent to NOT EXISTS(X relation Y)

  • Any X WHERE NOT X owned_by U means “entities that have no relation owned_by”.

  • Any X WHERE NOT X owned_by U, U login "syt" means “the entity have no relation owned_by with the user syt”. They may have a relation “owned_by” with another user.

In this clause, you can also use EXISTS when you want to know if some expression is true and do not need the complete set of elements that make it true. Testing for existence is much faster than fetching the complete set of results, especially when you think about using OR against several expressions. For instance if you want to retrieve versions which are in state “ready” or tagged by “priority”, you should write :

Any X ORDERBY PN,N
WHERE X num N, X version_of P, P name PN,
      EXISTS(X in_state S, S name "ready")
      OR EXISTS(T tags X, T name "priority")

not

Any X ORDERBY PN,N
WHERE X num N, X version_of P, P name PN,
      (X in_state S, S name "ready")
      OR (T tags X, T name "priority")

Both queries aren’t at all equivalent :

  • the former will retrieve all versions, then check for each one which are in the matching state of or tagged by the expected tag,

  • the later will retrieve all versions, state and tags (cartesian product!), compute join and then exclude each row which are in the matching state or tagged by the expected tag. This implies that you won’t get any result if the in_state or tag tables are empty (ie there is no such relation in the application). This is usually NOT what you want.

Another common case where you may want to use EXISTS is when you find yourself using DISTINCT at the beginning of your query to remove duplicate results. The typical case is when you have a multivalued relation such as Version version_of Project and you want to retrieve projects which have a version:

Any P WHERE V version_of P

will return each project number of versions times. So you may be tempted to use:

DISTINCT Any P WHERE V version_of P

This will work, but is not efficient, as it will use the SELECT DISTINCT SQL predicate, which needs to retrieve all projects, then sort them and discard duplicates, which can have a very high cost for large result sets. So the best way to write this is:

Any P WHERE EXISTS(V version_of P)

You can also use the question mark (?) to mark optional relations. This allows you to select entities related or not to another. It is a similar concept to Left outer join:

the result of a left outer join (or simply left join) for table A and B always contains all records of the “left” table (A), even if the join-condition does not find any matching record in the “right” table (B).

You must use the ? behind a variable to specify that the relation to that variable is optional. For instance:

  • Bugs of a project attached or not to a version

    Any X, V WHERE X concerns P, P eid 42, X corrected_in V?

    You will get a result set containing all the project’s tickets, with either the version in which it’s fixed or None for tickets not related to a version.

  • All cards and the project they document if any

    Any C, P WHERE C is Card, P? documented_by C

Notice you may also use outer join:

  • on the RHS of attribute relation, e.g.

    Any X WHERE X ref XR, Y name XR?

    so that Y is outer joined on X by ref/name attributes comparison

  • on any side of an HAVING expression, e.g.

    Any X WHERE X creation_date XC, Y creation_date YC
HAVING YEAR(XC)=YEAR(YC)?

    so that Y is outer joined on X by comparison of the year extracted from their creation date.

    Any X WHERE X creation_date XC, Y creation_date YC
HAVING YEAR(XC)?=YEAR(YC)

    would outer join X on Y instead.

Having restrictions#

The HAVING clause, as in SQL, may be used to restrict a query according to value returned by an aggregate function, e.g.

Any X GROUPBY X WHERE X relation Y HAVING COUNT(Y) > 10

It may however be used for something else: In the WHERE clause, we are limited to triplet expressions, so some things may not be expressed there. Let’s take an example : if you want to get people whose upper-cased first name equals to another person upper-cased first name. There is no proper way to express this using triplet, so you should use something like:

Any X WHERE X firstname XFN, Y firstname YFN, NOT X identity Y HAVING UPPER(XFN) = UPPER(YFN)

Another example: imagine you want person born in 2000:

Any X WHERE X birthday XB HAVING YEAR(XB) = 2000

Notice that while we would like this to work without the HAVING clause, this can’t be currently be done because it introduces an ambiguity in RQL’s grammar that can’t be handled by Yapps, the parser’s generator we’re using.

Sub-queries#

The WITH keyword introduce sub-queries clause. Each sub-query has the form:

V1(,V2) BEING (rql query)

Variables at the left of the BEING keyword defines into which variables results from the sub-query will be mapped to into the outer query. Sub-queries are separated from each other using a comma.

Let’s say we want to retrieve for each project its number of versions and its number of tickets. Due to the nature of relational algebra behind the scene, this can’t be achieved using a single query. You have to write something along the line of:

Any X, VC, TC WHERE X identity XX
WITH X, VC BEING (Any X, COUNT(V) GROUPBY X WHERE V version_of X),
     XX, TC BEING (Any X, COUNT(T) GROUPBY X WHERE T ticket_of X)

Notice that we can’t reuse a same variable name as alias for two different sub-queries, hence the usage of ‘X’ and ‘XX’ in this example, which are then unified using the special identity relation (see Virtual relations).

Warning

Sub-queries define a new variable scope, so even if a variable has the same name in the outer query and in the sub-query, they technically aren’t the same variable. So:

Any W, REF WITH W, REF BEING
    (Any W, REF WHERE W is Workcase, W ref REF,
                      W concerned_by D, D name "Logilab")

could be written:

Any W, REF WITH W, REF BEING
   (Any W1, REF1 WHERE W1 is Workcase, W1 ref REF1,
                       W1 concerned_by D, D name "Logilab")

Also, when a variable is coming from a sub-query, you currently can’t reference its attribute or inlined relations in the outer query, you’ve to fetch them in the sub-query. For instance, let’s say we want to sort by project name in our first example, we would have to write:

Any X, VC, TC ORDERBY XN WHERE X identity XX
WITH X, XN, VC BEING (Any X, COUNT(V) GROUPBY X,XN WHERE V version_of X, X name XN),
     XX, TC BEING (Any X, COUNT(T) GROUPBY X WHERE T ticket_of X)

instead of:

Any X, VC, TC ORDERBY XN WHERE X identity XX, X name XN,
WITH X, XN, VC BEING (Any X, COUNT(V) GROUPBY X WHERE V version_of X),
     XX, TC BEING (Any X, COUNT(T) GROUPBY X WHERE T ticket_of X)

which would result in a SQL execution error.

Union#

You may get a result set containing the concatenation of several queries using the UNION. The selection of each query should have the same number of columns.

(Any X, XN WHERE X is Person, X surname XN) UNION (Any X,XN WHERE X is Company, X name XN)

Available functions#

Below is the list of aggregate and transformation functions that are supported natively by the framework. Notice that cubes may define additional functions.

Aggregate functions#

COUNT(Any)

return the number of rows

MIN(Any)

return the minimum value

MAX(Any)

return the maximum value

AVG(Any)

return the average value

SUM(Any)

return the sum of values

GROUP_CONCAT(String)

return each unique value separated by a comma (for string only)

All aggregate functions above take a single argument. Take care some aggregate functions (e.g. MAX, MIN) may return None if there is no result row.

String transformation functions#

UPPER(String)

upper case the string

LOWER(String)

lower case the string

LENGTH(String)

return the length of the string

SUBSTRING(String, start, length)

extract from the string a string starting at given index and of given length

TEXT_LIMIT_SIZE(String, max size)

if the length of the string is greater than given max size, strip it and add ellipsis (”…”). The resulting string will hence have max size + 3 characters

LIMIT_SIZE(String, format, max size)

similar to the above, but allow to specify the MIME type of the text contained by the string. Supported formats are text/html, text/xhtml and text/xml. All others will be considered as plain text. For non plain text format, sgml tags will be first removed before limiting the string.
Date extraction functions#

YEAR(Date)

return the year of a date or datetime

MONTH(Date)

return the month of a date or datetime

DAY(Date)

return the day of a date or datetime

HOUR(Datetime)

return the hours of a datetime

MINUTE(Datetime)

return the minutes of a datetime

SECOND(Datetime)

return the seconds of a datetime

WEEKDAY(Date)

return the day of week of a date or datetime. Sunday == 1, Saturday == 7.
Other functions#

ABS(num)

return the absolute value of a number

RANDOM()

return a pseudo-random value from 0.0 to 1.0

FSPATH(X)

expect X to be an attribute whose value is stored in a BFSStorage and return its path on the file system

FTIRANK(X)

expect X to be an entity used in a has_text relation, and return a number corresponding to the rank order of each resulting entity

CAST(Type, X)

expect X to be an attribute and return it casted into the given final type

Examples#

  • Search for the object of identifier 53

    Any X WHERE X eid 53

  • Search material such as comics, owned by syt and available

    Any X WHERE X is Document,
            X occurence_of F, F class C, C name 'Comics',
            X owned_by U, U login 'syt',
            X available TRUE

  • Looking for people working for eurocopter interested in training

    Any P WHERE P is Person, P work_for S, S name 'Eurocopter',
            P interested_by T, T name 'training'

  • Search note less than 10 days old written by jphc or ocy

    Any N WHERE N is Note, N written_on D, D day> (today -10),
            N written_by P, P name 'jphc' or P name 'ocy'

  • Looking for people interested in training or living in Paris

    Any P WHERE P is Person, EXISTS(P interested_by T, T name 'training') OR
            (P city 'Paris')

  • The surname and firstname of all people

    Any N, P WHERE X is Person, X name N, X firstname P

    Note that the selection of several entities generally force the use of “Any” because the type specification applies otherwise to all the selected variables. We could write here

    String N, P WHERE X is Person, X name N, X first_name P

    Note: You can not specify several types with * … where X is FirstType or X is SecondType*. To specify several types explicitly, you have to do

    Any X WHERE X is IN (FirstType, SecondType)

Insertion query#

INSERT <entity type> V1 (, <entity type> V2) * : <assignments> [ WHERE <restriction>]

assignments

list of relations to assign in the form V1 relationship V2 | <static value>

The restriction can define variables used in assignments.

Caution, if a restriction is specified, the insertion is done for each line result returned by the restriction.

  • Insert a new person named ‘foo’

    INSERT Person X: X name 'foo'

  • Insert a new person named ‘foo’, another called ‘nice’ and a ‘friend’ relation between them

    INSERT Person X, Person Y: X name 'foo', Y name 'nice', X friend Y

  • Insert a new person named ‘foo’ and a ‘friend’ relation with an existing person called ‘nice’

    INSERT Person X: X name 'foo', X friend  Y WHERE Y name 'nice'

Update and relation creation queries#

SET <assignements> [ WHERE <restriction>]

Caution, if a restriction is specified, the update is done for each result line returned by the restriction.

  • Renaming of the person named ‘foo’ to ‘bar’ with the first name changed

    SET X name 'bar', X firstname 'original' WHERE X is Person, X name 'foo'

  • Insert a relation of type ‘know’ between objects linked by the relation of type ‘friend’

    SET X know Y  WHERE X friend Y

Deletion query#

DELETE (<entity type> V) | (V1 relation v2 ),… [ WHERE <restriction>]

Caution, if a restriction is specified, the deletion is made for each line result returned by the restriction.

  • Deletion of the person named ‘foo’

    DELETE Person X WHERE X name 'foo'

  • Removal of all relations of type ‘friend’ from the person named ‘foo’

    DELETE X friend Y WHERE X is Person, X name 'foo'

Debugging RQL#

Available levels#

Server debugging flags. They may be combined using binary operators.

cubicweb.server.DBG_NONE = 0#

no debug information

cubicweb.server.DBG_RQL = 1#

rql execution information

cubicweb.server.DBG_SQL = 2#

executed sql

cubicweb.server.DBG_REPO = 4#

repository events

cubicweb.server.DBG_HOOKS = 16#

hooks

cubicweb.server.DBG_OPS = 32#

operations

cubicweb.server.DBG_MORE = 128#

more verbosity

cubicweb.server.DBG_ALL = 247#

all level enabled

Enable verbose output#

To debug your RQL statements, it can be useful to enable a verbose output:

from cubicweb import server
server.set_debug(server.DBG_RQL|server.DBG_SQL|server.DBG_ALL)
cubicweb.server.set_debug(debugmode)[source]#

change the repository debugging mode

Another example showing how to debug hooks at a specific code site:

from cubicweb.server import debugged, DBG_HOOKS
with debugged(DBG_HOOKS):
    person.cw_set(works_for=company)

Detect largest RQL queries#

See Profiling and performance chapter (see Profiling and performance).

API#

class cubicweb.server.debugged(debugmode)[source]#

Context manager and decorator to help debug the repository.

It can be used either as a context manager:

>>> with debugged('DBG_RQL | DBG_REPO'):
...     # some code in which you want to debug repository activity,
...     # seing information about RQL being executed an repository events.

or as a function decorator:

>>> @debugged('DBG_RQL | DBG_REPO')
... def some_function():
...     # some code in which you want to debug repository activity,
...     # seing information about RQL being executed an repository events

The debug mode will be reset to its original value when leaving the “with” block or the decorated function.

RQL usecases#

Use of RQL in Card documents - ReST#

With a CubicWeb instance supporting object types with ReST content (for example Card), one can build content based on RQL queries as dynamic documents.

For this, use the rql and rql-table ReST directive, for more information about custom ReST directives head over to the sphinx documentation which uses them extensively.

rql directive#

The rql directive takes as input an RQL expression and a view to apply to the result.

For example, create a Card content by opening http://cubicweb_example.org/add/Card and add the following content, as an example : a table of blog entries (10 most recent blog entries table with user and date information)

Recent blog entries
-------------------

:rql:`Any B,U,D ORDERBY D DESC LIMIT 10 WHERE B is BlogEntry, B title T, B creation_date D, B created_by U:table`
book/images/example-card-with-rql-directive.png

rql-table directive#

rql-table enables more customization, enabling you to modify the column (header) contents, and the view applied for a specific column (colvids).

For example, create a Card content by opening http://cubicweb_example.org/add/Card and add the following content

Blog entries with rql-table
-----------------------------

.. rql-table::
   :vid: table
   :headers: Title with link, who wrote it, at what date
   :colvids: 1=sameetypelist

   Any B,U,D ORDERBY D DESC LIMIT 10 WHERE B is BlogEntry, B title T, B creation_date D, B created_by U

All fields but the RQL string are optional. The :headers: option can contain empty column names.

book/images/example-card-with-rql-table-directive.png

Use in python projects and CLI#

cwclientlib <https://pypi.org/project/cwclientlib/> enables you to use RQL in your python projects using only web requests. This project also provides a remote command line interface (CLI) you can use to replace a server side cubicweb-ctl shell.

Use in JavaScript/React components#

cwclientelements <https://forge.extranet.logilab.fr/open-source/cwclientelements> is a library of reusable React components for building web application with cubicweb and RQL.

Implementation#

BNF grammar#

The terminal elements are in capital letters, non-terminal in lowercase. The value of the terminal elements (between quotes) is a Python regular expression.

statement ::= (select | delete | insert | update) ';'


# select specific rules
select      ::= 'DISTINCT'? E_TYPE selected_terms restriction? group? sort?

selected_terms ::= expression ( ',' expression)*

group       ::= 'GROUPBY' VARIABLE ( ',' VARIABLE)*

sort        ::= 'ORDERBY' sort_term ( ',' sort_term)*

sort_term   ::=  VARIABLE sort_method =?

sort_method ::= 'ASC' | 'DESC'


# delete specific rules
delete ::= 'DELETE' (variables_declaration | relations_declaration) restriction?


# insert specific rules
insert ::= 'INSERT' variables_declaration ( ':' relations_declaration)? restriction?


# update specific rules
update ::= 'SET' relations_declaration restriction


# common rules
variables_declaration ::= E_TYPE VARIABLE (',' E_TYPE VARIABLE)*

relations_declaration ::= simple_relation (',' simple_relation)*

simple_relation ::= VARIABLE R_TYPE expression

restriction ::= 'WHERE' relations

relations   ::= relation (LOGIC_OP relation)*
              | '(' relations')'

relation    ::= 'NOT'? VARIABLE R_TYPE COMP_OP? expression
              | 'NOT'? R_TYPE VARIABLE 'IN' '(' expression (',' expression)* ')'

expression  ::= var_or_func_or_const (MATH_OP var_or_func_or_const) *
              | '(' expression ')'

var_or_func_or_const ::= VARIABLE | function | constant

function    ::= FUNCTION '(' expression ( ',' expression) * ')'

constant    ::= KEYWORD | STRING | FLOAT | INT

# tokens
LOGIC_OP ::= ',' | 'OR' | 'AND'
MATH_OP  ::= '+' | '-' | '/' | '*'
COMP_OP  ::= '>' | '>=' | '=' | '<=' | '<' | '~=' | 'LIKE'

FUNCTION ::= 'MIN' | 'MAX' | 'SUM' | 'AVG' | 'COUNT' | 'UPPER' | 'LOWER'

VARIABLE ::= '[A-Z][A-Z0-9]*'
E_TYPE   ::= '[A-Z]\w*'
R_TYPE   ::= '[a-z_]+'

KEYWORD  ::= 'TRUE' | 'FALSE' | 'NULL' | 'TODAY' | 'NOW'
STRING   ::= "'([^'\]|\\.)*'" |'"([^\"]|\\.)*\"'
FLOAT    ::= '\d+\.\d*'
INT      ::= '\d+'

Internal representation (syntactic tree)#

The tree research does not contain the selected variables (e.g. there is only what follows “WHERE”).

The insertion tree does not contain the variables inserted or relations defined on these variables (e.g. there is only what follows “WHERE”).

The removal tree does not contain the deleted variables and relations (e.g. there is only what follows the “WHERE”).

The update tree does not contain the variables and relations updated (e.g. there is only what follows the “WHERE”).

Select         ((Relationship | And | Or)?, Group?, Sort?)
Insert         (Relations | And | Or)?
Delete         (Relationship | And | Or)?
Update         (Relations | And | Or)?

And            ((Relationship | And | Or), (Relationship | And | Or))
Or             ((Relationship | And | Or), (Relationship | And | Or))

Relationship   ((VariableRef, Comparison))

Comparison     ((Function | MathExpression | Keyword | Constant | VariableRef) +)

Function       (())
MathExpression ((MathExpression | Keyword | Constant | VariableRef), (MathExpression | Keyword | Constant | VariableRef))

Group          (VariableRef +)
Sort           (SortTerm +)
SortTerm       (VariableRef +)

VariableRef    ()
Variable       ()
Keyword        ()
Constant       ()

Known limitations#

  • The current implementation does not support linking two relations of type ‘is’ with an OR. I do not think that the negation is supported on this type of relation (XXX to be confirmed).

  • missing COALESCE and certainly other things…

  • writing an rql query requires knowledge of the used schema (with real relation names and entities, not those viewed in the user interface). On the other hand, we cannot really bypass that, and it is the job of a user interface to hide the RQL.

Topics#

It would be convenient to express the schema matching relations (non-recursive rules):

Document class Type <-> Document occurence_of Fiche class Type
Sheet class Type    <-> Form collection Collection class Type

Therefore 1. becomes:

Document X where
X class C, C name 'Cartoon'
X owned_by U, U login 'syt'
X available true

I’m not sure that we should handle this at RQL level …

There should also be a special relation ‘anonymous’.

Introducing Mercurial#

Introduction#

Mercurial manages a distributed repository containing revisions trees (each revision indicates the changes required to obtain the next, and so on). Locally, we have a repository containing revisions tree, and a working directory. It is possible to put in its working directory, one of the versions of its local repository, modify and then push it in its repository. It is also possible to get revisions from another repository or to export its own revisions from the local repository to another repository.

In contrast to CVS/Subversion, we usually create a repository per project to manage.

In a collaborative development, we usually create a central repository accessible to all developers of the project. These central repository is used as a reference. According to their needs, everyone can have a local repository, that they will have to synchronize with the central repository from time to time.

Major commands#

  • Create a local repository:

    hg clone ssh://myhost//home/src/repo

  • See the contents of the local repository (graphical tool in Qt):

    hgview

  • Add a sub-directory or file in the current directory:

    hg add subdir

  • Move to the working directory a specific revision (or last revision) from the local repository:

    hg update [identifier-revision]
hg up [identifier-revision]

  • Get in its local repository, the tree of revisions contained in a remote repository (this does not change the local directory):

    hg pull ssh://myhost//home/src/repo
hg pull -u ssh://myhost//home/src/repo # equivalent to pull + update

  • See what are the heads of branches of the local repository if a pull returned a new branch:

    hg heads

  • Submit the working directory in the local repository (and create a new revision):

    hg commit
hg ci

  • Merge with the mother revision of local directory, another revision from the local respository (the new revision will be then two mothers revisions):

    hg merge identifier-revision

  • Export to a remote repository, the tree of revisions in its content local respository (this does not change the local directory):

    hg push ssh://myhost//home/src/repo

  • See what local revisions are not in another repository:

    hg outgoing ssh://myhost//home/src/repo

  • See what are the revisions of a repository not found locally:

    hg incoming ssh://myhost//home/src/repo

  • See what is the revision of the local repository which has been taken out from the working directory and amended:

    hg parent

  • See the differences between the working directory and the mother revision of the local repository, possibly to submit them in the local repository:

    hg diff
hg commit-tool
hg ct

Best Practices#

  • Remember to hg pull -u regularly, and particularly before

    a hg commit.

  • Remember to hg push when your repository contains a version relatively stable of your changes.

  • If a hg pull -u created a new branch head:

    1. find its identifier with hg head

    2. merge with hg merge

    3. hg ci

    4. hg push

More information#

For more information about Mercurial, please refer to the Mercurial project online documentation.

Installation dependencies#

When you run CubicWeb from source, either by downloading the tarball or cloning the mercurial tree, here is the list of tools and libraries you need to have installed in order for CubicWeb to work:

If you’re using a Postgresql database (recommended):

Other optional packages:

Any help with the packaging of CubicWeb for more than Debian/Ubuntu (including eggs, buildouts, etc) will be greatly appreciated.

Javascript docstrings#

Whereas in Python source code we only need to include a module docstrings using the directive .. automodule:: mypythonmodule, we will have to explicitely define Javascript modules and functions in the doctrings since there is no native directive to include Javascript files.

Rest generation#

pyjsrest is a small utility parsing Javascript doctrings and generating the corresponding Restructured file used by Sphinx to generate HTML documentation. This script will have the following structure:

===========
filename.js
===========
.. module:: filename.js

We use the .. module:: directive to register a javascript library as a Python module for Sphinx. This provides an entry in the module index.

The contents of the docstring found in the javascript file will be added as is following the module declaration. No treatment will be done on the doctring. All the documentation structure will be in the docstrings and will comply with the following rules.

Docstring structure#

Basically we document javascript with RestructuredText docstring following the same convention as documenting Python code.

The doctring in Javascript files must be contained in standard Javascript comment signs, starting with /** and ending with */, such as:

/**
 * My comment starts here.
 * This is the second line prefixed with a `*`.
 * ...
 * ...
 * All the follwing line will be prefixed with a `*` followed by a space.
 * ...
 * ...
 */

Comments line prefixed by // will be ignored. They are reserved for source code comments dedicated to developers.

Javscript functions docstring#

By default, the function directive describes a module-level function.

function directive#

Its purpose is to define the function prototype such as:

.. function:: loadxhtml(url, data, reqtype, mode)

If any namespace is used, we should add it in the prototype for now, until we define an appropriate directive:

.. function:: jQuery.fn.loadxhtml(url, data, reqtype, mode)

Function parameters#

We will define function parameters as a bulleted list, where the parameter name will be backquoted and followed by its description.

Example of a javascript function docstring:

.. function:: loadxhtml(url, data, reqtype, mode)

cubicweb loadxhtml plugin to make jquery handle xhtml response

fetches `url` and replaces this's content with the result

Its arguments are:

* `url`

* `mode`, how the replacement should be done (default is 'replace')
   Possible values are :
       - 'replace' to replace the node's content with the generated HTML
       - 'swap' to replace the node itself with the generated HTML
       - 'append' to append the generated HTML to the node's content

Optional parameter specification#

Javascript functions handle arguments not listed in the function signature. In the javascript code, they will be flagged using /* … */. In the docstring, we flag those optional arguments the same way we would define it in Python:

.. function:: asyncRemoteExec(fname, arg1=None, arg2=None)

Changelog#

3.38.7 (2023-03-07)#

👷 Bug fixes#

  • rdf: https instead of http for schema.org

  • sphinx-theme 1.0 breaks doc build

  • make sure we only install yapps2-logilab by updating depencies

  • tried to format a string while missing one formatting argument

3.38.6 (2023-02-13)#

👷 Bug fixes#

  • hooks: notification things are no more in “views” registry

3.38.5 (2023-01-31)#

👷 Bug fixes#

  • remove deprecated import to cubicweb.web

3.38.4 (2023-01-17)#

🎉 New features#

  • skeleton: remove format=pylint option from tox because it’s better without it

3.38.3 (2023-01-12)#

👷 Bug fixes#

🤷 Various changes#

  • changelog/3.38: add instruction on how to use 3.38/cubicweb_web_imports.py

3.38.2 (2023-01-03)#

👷 Bug fixes#

  • sobjects/notifications: keep RecipientsFinder class and subclass in the components registry for retrocompatibility

3.38.1 (2022-12-05)#

🎉 New features#

  • schema: Export relations options on the schema (merge from 3.37)

👷 Bug fixes#

  • schema_exporters: Add missing description field for relations (e.g in_state) to schema exporter (merge from 3.37)

3.38.0 (2022-11-22)#

This is the last major release of the 3.* before the 4 branch.

In this release the whole cubicweb.web module and the cubicweb/view.py file have been extracted in the cubicweb_web cube which is a dependency of cubicweb now. Automatic backward compatibility is provided by imports so your projects should work with this new version without modifications.

A script to help migrating to this version is available in this repository in the 3.38 folder https://forge.extranet.logilab.fr/cubicweb/cw_versions_migration_tools This script will change all the imports to match the news one for CubicWeb 3.38 and the cube cubicweb_web. It will not change your dependencies in your setup.py or __pkginfo__.py, you have to do this yourself.

Its usage, once the dependencies has been installed (only RedBaron), is the following:

python 3.38/cubicweb_web_imports.py <path to my project>

It will hopefully save you quite some time.

🎉 New features#

  • the cubicweb_web cube is now a dependency of cubicweb

  • add adapter_regid as parameter on add_entity_to_graph (#535)

  • add relation constraints to schema export

  • cubicweb.web extraction: change all import of cubicweb.web to cubicweb_web

  • cubicweb.web removal: add deprecation warning in view module

  • cubicweb.web removal: add generic deprecation warning in all web modules

  • cwctl: don’t check if we need to upgrade anything when running cwctl versions (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/563)

  • doc: clarify when rich had been removed

  • export relation options in schema options key

  • pkg: upgrade version of waitress to 2.1.1 or more, for security reason. (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/543)

  • rdf: use entity.absolute_url instead of cwuri in RDF adapters (#534)

  • redirection: pyramid redirection now keep parameters by default (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/566)

  • rql: Add “IRQLInterface” adapter to force defining a rql interface which is available on RQL projection varaibles

  • rql: Add entities function and attribute from RQL queries

  • serverctl: add a command to list all unused indexes

  • test/content-negociation: display rdf body on failing tests for easier debugging

  • test: use testing.cubicweb instead of testing.fr in test (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/374) BREAKING CHANGE: use testing.cubicweb instead of testing.fr in test

👷 Bug fixes#

  • add retrocompatibility for anonymized_request that is now in cubicweb_web

  • base64.decodestring is deprecated and has been removed

  • cubicweb_web/deprecations: increase warning stack level to show correct line

  • cubicweb_web: change magic modules imports to uses cubicweb_web

  • cwconfig: Ensure the cube web is available with cubicweb-ctl commands

  • CWRelation.rtype api is different from CWRelation.relation_type api

  • ensure that the “web” cube is in the list of cubes dependencies

  • hook: Search the notification view from the good registry

  • htmlwidgets: BoxLink rendering is broken

  • make i18ncube load web cube’s appobjects

  • mod: Load sobjects.notification and sobjects.supervising even if no cubicweb_web

  • notification: Make NotificationView inherits from AppObject

  • pyramid: adapt TestApp.post_json method to CSRF

  • pyramid: adapt TestApp.put_json method to CSRF

  • pyramid: try to get “/login” if “/” is forbidden

  • req: add missing set_log_methods on CubicWebRequestBase

  • schema_exporters: handle symmetrical relation in schema export. (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/568)

  • supervising: Adapt SupervisingView for the NotificationView API

  • supervising: Uses the NotificationView for the supervising instead of component

  • test-instance-creation: cubicweb now needs the web cube to be installed

🤖 Continuous integration#

Most python test have been splitted to speed up the CI speed.

  • .gitlab-ci.yml: refactoring py3 tests declaration using a base template

  • add check-dependencies-resolution job

  • add mypy job

  • add safety job

  • add twine-check job

  • fix: “base” in py3-server-base clashed with “py3-base”, use “core” instead

  • fix: py3-auto-test-views jobs wrongly launched py3-server-bases tests

  • migrate to v2 of templates

  • move to bullseye and pg13

  • split py3-misc into several different tests

  • split py3-server into several different tests

  • test-instance-creation: pip –use-feature=in-tree-build is deprecated, remove it

  • use .retry base template in (nearly) all jobs

🤷 Various changes#

  • [cubicweb 3.38] RequestSessionBase is deprecated, use RequestSessionAndConnectionBase instead

  • remove mailing-list from “how to contribute” since it’s no more used (https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/395)

  • supervising: Refactor to not using self.w from NotificationView

  • Unknown config option: log_print

3.37.14 (2023-03-07)#

👷 Bug fixes#

  • rdf: https instead of http for schema.org

  • sphinx-theme 1.0 breaks doc build

  • make sure we only install yapps2-logilab by updating depencies

  • tried to format a string while missing one formatting argument

3.37.13 (2023-01-27)#

🎉 New features#

  • misc: allow to disable or not constraint check on fast drop script

3.37.12 (2023-01-17)#

🎉 New features#

  • skeleton: remove format=pylint option from tox because it’s better without it

3.37.11 (2023-01-12)#

👷 Bug fixes#

3.37.10 (2022-12-05)#

🎉 New features#

  • schema: Export relations options on the schema

👷 Bug fixes#

  • schema_exporters: Add missing description field for relations (e.g in_state) to schema exporter

3.37.9 (2022-11-15)#

👷 Bug fixes#

  • hook: correct a typo, self.warn doesn’t exist

3.37.8 (2022-10-04)#

👷 Bug fixes#

  • attr: when an entity is not existing always return None when fetching its attributes (#599)

  • web.views: escape text from the undohistory view (#598)

🤷 Various changes#

  • delete unused translations from *.po files (#600)

  • skeleton: add long_description_content_type in setup.py

3.37.7 (2022-09-22)#

👷 Bug fixes#

3.37.6 (2022-09-14)#

👷 Bug fixes#

  • bookmark: do not escape the xaddrelation view from ajaxedit module

3.37.5 (2022-08-30)#

👷 Bug fixes#

  • pyramid: Redirect to the wanted URL after a successfully loggedin (to #584)

  • xss: Ensure to use the xml_escape method on entity attributes

  • perf: Restore initial performances by removing the uneccessary join

3.37.4 (2022-07-21)#

👷 Bug fixes#

🤷 Various changes#

3.37.3 (2022-07-13)#

👷 Bug fixes#

  • htmlwidgets: BoxLink rendering is broken

3.37.2 (2022-06-03)#

👷 Bug fixes#

  • pyramid: adapt TestApp.put_json method to CSRF

3.37.1 (2022-06-01)#

🎉 New features#

👷 Bug fixes#

  • base64.decodestring is deprecated and has been removed

  • pyramid tests: adapt TestApp.post_json method to CSRF

  • pyramid tests: try to get “/login” if “/” is forbidden

3.37.0 (2022-03-31)#

Breaking changes#

  • cubicweb.web.BaseWebConfiguration and cubicweb.web.WebConfigurationBase have been merged into cubicweb.web.WebConfiguration

  • cubicweb.web.CubicWebPyramidConfiguration had been removed

  • you can nomore use -c option when creating a CW instance, since there is now only one kind of configuration: all-in-one.conf

🎉 New features#

  • add attributes constraints in exported schema

  • depends on yams 0.48

  • doc: mostly add links of issues

👷 Bug fixes#

  • unittest_devctl: give all debugging informations

🤖 Continuous integration#

  • use templates

🤷 Various changes#

  • refactor!: merge BaseWebConfiguration into WebConfiguration

  • refactor!: remove -c option to cubicweb-ctl create to only use all-in-one

  • refactor!: remove unused CubicWebPyramidConfiguration

3.36.14 (2023-03-02)#

👷 Bug fixes#

  • sphinx-theme 1.0 breaks doc build

3.36.13 (2023-03-02)#

👷 Bug fixes#

  • make sure we only install yapps2-logilab by updating depencies

  • tried to format a string while missing one formatting argument

3.36.12 (2023-01-17)#

🎉 New features#

  • skeleton: remove format=pylint option from tox because it’s better without it

3.36.11 (2023-01-12)#

👷 Bug fixes#

3.36.10 (2022-11-15)#

👷 Bug fixes#

  • hook: correct a typo, self.warn doesn’t exist

3.36.9 (2022-10-04)#

👷 Bug fixes#

  • attr: when an entity is not existing always return None when fetching its attributes (#599)

  • web.views: escape text from the undohistory view (#598)

🤷 Various changes#

  • delete unused translations from *.po files (#600)

  • skeleton: add long_description_content_type in setup.py

3.36.8 (2022-09-22)#

👷 Bug fixes#

3.36.7 (2022-09-14)#

👷 Bug fixes#

  • bookmark: do not escape the xaddrelation view from ajaxedit module

3.36.6 (2022-08-30)#

👷 Bug fixes#

  • pyramid: Redirect to the wanted URL after a successfully loggedin (to #584)

  • xss: Ensure to use the xml_escape method on entity attributes

  • perf: Restore initial performances by removing the uneccessary join

3.36.5 (2022-07-21)#

👷 Bug fixes#

🤷 Various changes#

3.36.4 (2022-07-13)#

  • merge 3.35.6 into 3.36

3.36.3 (2022-06-03)#

👷 Bug fixes#

  • pyramid: adapt TestApp.put_json method to CSRF

3.36.2 (2022-06-01)#

🎉 New features#

👷 Bug fixes#

  • base64.decodestring is deprecated and has been removed

  • pyramid: adapt TestApp.post_json method to CSRF

  • pyramid: try to get “/login” if “/” is forbidden

3.36.1 (2022-03-31)#

👷 Bug fixes#

  • rql2sql: upgrade RQL version to fix translation of NOT EXISTS(X eid Y) (#528)

  • view: don’t escape html tags inside image previews

3.36.0 (2022-03-14)#

🎉 New features#

  • markdown: load extra extensions to render tables (#515)

  • schema_exporter: add a parameter to export schema as dict (#522)

📝 Documentation#

  • fix sidebar table of content

  • improve basic tutorial

  • improve home and sidebar

  • improve setup instructions

  • improve skeleton readme

  • set version number

  • use relative links for static resources

  • use right number of characters for titles

  • use sphinx_book_theme

3.35.12 (2022-11-15)#

👷 Bug fixes#

  • hook: correct a typo, self.warn doesn’t exist

3.35.11 (2022-10-04)#

👷 Bug fixes#

  • attr: when an entity is not existing always return None when fetching its attributes (#599)

  • web.views: escape text from the undohistory view (#598)

🤷 Various changes#

  • delete unused translations from *.po files (#600)

  • skeleton: add long_description_content_type in setup.py

3.35.10 (2022-09-22)#

👷 Bug fixes#

3.35.9 (2022-09-14)#

👷 Bug fixes#

  • bookmark: do not escape the xaddrelation view from ajaxedit module

3.35.8 (2022-08-30)#

👷 Bug fixes#

  • pyramid: Redirect to the wanted URL after a successfully loggedin (to #584)

  • xss: Ensure to use the xml_escape method on entity attributes

  • perf: Restore initial performances by removing the uneccessary join

3.35.7 (2022-07-21)#

👷 Bug fixes#

🤷 Various changes#

3.35.6 (2022-07-13)#

🤷 Various changes#

  • fix warnings of yams 0.48+ (3.35 requires <0.48)

3.35.5 (2022-07-13)#

👷 Bug fixes#

  • basecontrollers: str object have no more “decode” method since py3

  • fix some warnings of yams 0.48+

  • htmlwidgets: BoxLink rendering is broken

  • server: remove a memory leak related to a file

  • test: improve one related to CSRF

3.35.4 (2022-06-03)#

👷 Bug fixes#

  • pyramid: adapt TestApp.put_json method to CSRF

3.35.3 (2022-06-01)#

🎉 New features#

👷 Bug fixes#

  • base64.decodestring is deprecated and has been removed

  • pyramid: adapt TestApp.post_json method to CSRF

  • pyramid: try to get “/login” if “/” is forbidden

3.35.2 (2022-03-31)#

👷 Bug fixes#

  • rql2sql: upgrade RQL version to fix translation of NOT EXISTS(X eid Y) (#528)

  • view: don’t escape html tags inside image previews

3.35.1 (2022-03-09)#

  • avoid excaping cubicweb:loadurl’s value twice (to #523)

  • don’t escape whole key=”value” attributes in TreeViewItemView (to #523)

  • escape URLs passed as href attributes (to #523)

3.35 (2022-02-02)#

Breaking changes#

  • deprecate RQLSuggestionsBuilder component ; users of this component should now use rqlsuggestions.RQLSuggestionsBuilder instead. RQL bar completion behaviour can be changed by replacing the “rql_suggest” ajax function. If this function isn’t registered, rql completion is disabled.

  • remove RQLNoSuggestionsBuilder

  • disable login using GET requests for security reasons

  • web: remove support of old Internet Explorer versions: add_css no longer accepts iespec and ieonly arguments

🎉 New features#

  • add a Dockerfile in the skeleton

  • add a function for deleting entities faster

  • config: add help messages in configuration files (all-in-one and sources)

  • disable constraints checks on the DB upon deletion

  • show cube name when there is a version conflict

  • skeleton: add release-new in skeleton

  • upgrade to yams 0.47

  • content negociation: we now can use /<etype>/<rest_attr> route for content negociation, if rest_attr is defined, the route /<etype>/<rest_attr> is disabled for content negociation in this situation

👷 Bug fixes#

  • relation_type not existing in some conditions on RelationDefinition (ionDefinition.rtype has been deprecated in yams in favor of relation_type)

🤖 Continuous integration#

  • only collect warnings when running tests on the default branch (#489)

3.34.3 (2022-03-31)#

👷 Bug fixes#

  • rql2sql: upgrade RQL version to fix translation of NOT EXISTS(X eid Y) (#528)

  • view: don’t escape html tags inside image previews

3.34.2 (2022-03-09)#

👷 Bug fixes#

  • avoid excaping cubicweb:loadurl’s value twice (to #523)

  • don’t escape whole key=”value” attributes in TreeViewItemView (to #523)

  • escape URLs passed as href attributes (to #523)

3.34.1 (2021-12-01)#

👷 Bug fixes#

  • server: correct RQL generation when we have function in ORDERBY (#466)

3.34.0 (2021-11-23)#

Breaking changes#

  • Python 3.7 is now the minimum supported version of Python;

  • test: settings = {"cubicweb.bwcompat": true} is now the default for test, please, check your test if they are failing because of this;

  • test: the qunit test driver has been removed;

  • remove our deprecated and unused wsgi module;

  • fix!(handler): rediction to login on cubicweb.AuthenticationError Previously we were sending a forbidden response (403) with the login form as the html content, now we redirect (303) to the login form instead;

  • rich <https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/348>, introduced in 3.33 for nice tracebacks, had been removed <https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/434#note_93131>.

Since we updated rdflib to version 6, some packages like rdflib_jsonld are no longuer needed. Please, check your dependencies if you have any issue.

🎉 New features#

  • add an export-schema command to cwctl

  • csrf: add debug login when creating a new csrf token

  • get_cleaned_form_data: add backward deprecated compatibility on req.form

  • pyramid/test/ux: better debugging information when failing to get CSRF token

  • security: implement inforcing form validation for POST arguments

  • store: allow stores to be used as context manager (#446)

👷 Bug fixes#

  • allow in Int as rest_attr

  • build_doc: docutils version 0.18.0 breaks doc building

  • doc8: indentation was using tabs in 3.32_reledit.rst

  • ldap: upgrade to ldap3 (datetime, encode fix)

  • p3-misc: missing fyzz modules for certains tests in spa2rql

  • pyramid/test: webapp handles cookies for us, we don’t need to manually set them

  • RDFLib: Remove rdflib-jsonld dependency and use RDFLib v6 jsonld builtin parser BREAKING CHANGE: The RDFLib v6 does not support python 3.6 anymore. With this dependency, CubicWeb neither.

  • reledit: Do not retrieve a list of schemata with _compute_ttypes

  • remove qunit test stuff (#447)

  • skeleton: use forge.extranet.logilab.fr as default web url for new cubes (#463)

  • startup: Fix RQL query to take advantage of caching (#384)

  • store csrf token during login

  • test: make anonymous user creation hook tests pass (#452)

  • utils: remove an useless space character

  • views: remove unneeded xml_escape for primary titles

🤖 Continuous integration#

  • allow sonarqube to fails until we fix the internal url problem

  • test: don’t wait lint to run tests (#445)

  • use some gitlab-ci-templates (#455)

  • uses buster-slim-pg11-firefox custom image for py3-auto-test job

  • uses cubicweb/dockerfiles/can-i-merge image to optimize can-i-merge job

🤷 Various changes#

  • instance-config: add attributes for authenticated smtp

  • pyramid.compat is deprecated and will be removed in Pyramid 2.0. The functionality is no longer necessary, as Pyramid 2.0 drops support for Python 2.

  • webconfig: remove an unused configuration option

3.33.13 (2022-03-09)#

👷 Bug fixes#

  • avoid excaping cubicweb:loadurl’s value twice (to #523)

  • don’t escape whole key=”value” attributes in TreeViewItemView (to #523)

  • escape URLs passed as href attributes (to #523)

3.33.12 (2021-12-01)#

👷 Bug fixes#

  • server: correct RQL generation when we have function in ORDERBY (#466)

3.33.11 (2021-11-17)#

👷 Bug fixes#

  • pkg: pin Yams version < 0.46.0

3.33.10 (2021-11-17)#

  • Removed allowed-http-host-headers configuration (which was a breaking change), since we don’t have this vulnerability in CubicWeb.

Various changes#

  • depend on sphinx >= 4.3

3.33.9 (2021-11-08)#

👷 Bug fixes#

  • views: remove an abusive escape (#457)

3.33.8 (2021-11-02)#

👷 Bug fixes#

  • build_doc: docutils version 0.18.0 breaks doc building (#443)

  • pkg: don’t use pyparsing 3 since it’s not compatible with rdflib 5 (#441)

  • pkg: pin pyramid_multiauth version to avoid compatibility issue with Pyramid 1 (#450)

3.33.7 (2021-10-12)#

👷 Bug fixes#

  • ldap: upgrade to ldap3 (datetime, encode fix)

  • csrf: ensure that we have a csrf token returned on every requests

3.33.6 (2021-10-04)#

👷 Bug fixes#

  • facet: remove abusive escaping in facets views. (#394)

3.33.5 (2021-09-29)#

👷 Bug fixes#

  • backout “limit setuptools version to avoid issue with 2to3”

  • use our package rdflib-jsonld-without-2to3, this is a fork of rdflib-jsonld with 2to3 usage removed, but which still contains the whole package code unlike rdflib-jsonld 0.6.x.

  • reledit: Do not retrieve a list of schemata with _compute_ttypes

3.33.4 (2021-09-24)#

  • don’t escape value in navigation components (#389)

  • views: remove unneeded xml_escape for primary titles

  • setup: keep rdflib-jsonld at version < 0.6.0

  • setup: limit setuptools version to avoid issue with 2to3

3.33.3 (2021-09-14)#

  • upgrade rdflib-jsonld version to keep compatibility with setupools 58 and above

👷 Bug fixes#

  • startup: Fix RQL query to take advantage of caching (#384)

3.33.2 (2021-09-02)#

📝 Documentation#

  • tuto: Fix path

3.33.1 (2021-08-31)#

🎉 New features#

  • allowed-http-host-headers: automatically add default hostname to the allowed list on debug mode

  • req: Add a “limit” parameter to RequestSessionBase.find

  • req: Add exists for optimized search of at least one entity

  • ux: better error message when a controller can’t be select

👷 Bug fixes#

  • fyzz dep was missing for running certain tests

  • only fyzz 0.2.2 is compatible with python 3

  • typo: fix some mispellings

📝 Documentation#

  • fix allowed-http-host-header label and quote from Django’s doc

🤖 Continuous integration#

  • integrate can-i-merge

🤷 Various changes#

  • 3.33: improve changelog quality

  • fix(bwcompat)!: return a 400 instead of a 401 when failed to select a controller

  • fix: allowed-http-host-headers has been released in 3.33 actually

  • misc: fix rst syntax

3.33.0 (2021-08-03)#

🎉 New features#

  • BREAKING security: introduce allowed-http-host-header against host attack (However, this is backed out in 3.33.10).

  • add postgresql extra requires

  • config: add ‘debug’ option in “[main]” of all-in-one.conf that does the same thing than “-D” in “cubicweb-ctl pyramid”

  • rich <https://github.com/willmcgugan/rich/>: to have nicer tracebacks, use rich.traceback <https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/348> (removed in 3.34 <https://forge.extranet.logilab.fr/cubicweb/cubicweb/-/issues/434#note_93131>)

👷 Bug fixes#

  • add default value for params argument in pyramid webtest post function (#350)

  • csrf: give CSRF token when using /ajax route

  • empty identification cookie on webapp.reset()

  • pin rdflib < 6.0.0 to avoid compatibility issues

  • rdf: graph.serialize needs to encode its content in utf-8

  • security: change configuration [WEB]interface default value to 127.0.0.1

  • views: Fix reledit errors when trying modify relation with multi subjects

🤖 Continuous integration#

  • use image from heptapod registry since r.intra was shut down

🤷 Various changes#

  • use open-source/gitlab-ci-templates in cube skeleton

  • drop mention of MySQL and SQLServer support

  • update cube installation procedure documentation

  • remove *.spec from skeleton

3.32.14 (2021-12-01)#

👷 Bug fixes#

  • server: correct RQL generation when we have function in ORDERBY (#466)

3.32.13 (2021-11-17)#

👷 Bug fixes#

  • pkg: pin Yams version < 0.46.0

3.32.12 (2021-11-17)#

Various changes#

  • depend on sphinx>=4.3

3.32.11 (2021-11-08)#

👷 Bug fixes#

  • views: remove an abusive escape (#457)

3.32.10 (2021-11-02)#

👷 Bug fixes#

  • build_doc: docutils version 0.18.0 breaks doc building (#443)

  • pkg: don’t use pyparsing 3 since it’s not compatible with rdflib 5 (#441)

  • pkg: pin pyramid_multiauth version to avoid compatibility issue with Pyramid 1 (#450)

3.32.9 (2021-10-12)#

👷 Bug fixes#

  • csrf: ensure that we have a csrf token returned on every requests

3.32.8 (2021-10-04)#

👷 Bug fixes#

  • facet: remove abusive escaping in facets views (#394)

3.32.7 (2021-09-29)#

👷 Bug fixes#

  • backout “limit setuptools version to avoid issue with 2to3”

  • use our package rdflib-jsonld-without-2to3, this is a fork of rdflib-jsonld with 2to3 usage removed, but which still contains the whole package code unlike rdflib-jsonld 0.6.x.

  • reledit: Do not retrieve a list of schemata with _compute_ttypes

3.32.6 (2021-09-24)#

👷 Bug fixes#

  • don’t escape value in navigation components (#398)

  • views: remove unneeded xml_escape for primary titles

  • setup: keep rdflib-jsonld at version < 0.6.0

  • setup: limit setuptools version to avoid issue with 2to3

3.32.5 (2021-09-14)#

  • upgrade rdflib-jsonld version to keep compatibility with setupools 58 and above

3.32.4 (2021-09-02)#

👷 Bug fixes#

  • do not use localhost.local has test domain, but keep the one already defined

3.32.3 (2021-08-31)#

🎉 New features#

  • migration: add a migration script to warn about incompatibility of cwtags. (#367)

👷 Bug fixes#

  • bringing back CubicWebServerTC and porting it to pyramid

  • fix bad escaped values in web views

  • pkg: since we added csrf mecanism, we need pyramid >= 1.9

  • test_newcube were broken because we removed cubicweb-*.spec file but didn’t updated the tests

3.32.2 (2021-07-30)#

🎉 New features#

  • use open-source/gitlab-ci-templates in cube skeleton

👷 Bug fixes#

  • add default value for params argument of PyramidCWTest.webapp.post (#350)

  • csrf: give CSRF token when using /ajax route

  • empty identification cookie on webapp.reset()

  • remove *.spec from skeleton

  • views: Fix reledit errors when trying modify relation with multi subjects

3.32.1 (2021-07-23)#

👷 Bug fixes#

  • pin rdflib < 6.0.0 to avoid compatibility issues

🤖 Continuous integration#

  • use image from heptapod registry since r.intra was shut down

3.32.0 (2021-07-13)#

🔒 Security, breaking changes#

Protection against XSS#

self.w API has been changed to automatically escape arguments used to format the string to mitigate XSS attacks.

This means that instead of writing:

self.w("some %s string %s" % (a, b))

You need to write:

self.w("some %s string %s", a, b)

And CubicWeb will escape all arguments given to self.w which are a and b here.

If for a specific reason (for example generating javascript) you don’t want to escape the arguments of self.w you can use the escape kwarg argument of self.w like this:

self.w("some %s string %s", a, b, escape=False)

This is normally retrocompatible since self.w old API with only one argument still works (but you shouldn’t use it anymore) but if you have been giving a custom function as self.w you’ll need to adapt the API of this function to match self.w new API which is:

def w(self, string, *args, **kwargs, escape=False): ...

Also note that UStringIO.write function has also been modified to be compatible with self.w new API (so if you are using it you won’t need to port this code).

CSRF protection#

A CSRF protection mechanism has been integrated in CubicWeb using Pyramid CSRF built-in protection. Regarding breaking changes:

  • Cubicweb now only works with pyramid

  • if you are only using cubicweb “web” without ajax and you have been doing advanced modification at the session management level this shouldn’t break anything for you

  • if you are doing POST/PUT/DELETE… requests using AJAX, you need to adapt your code to send the csrf_token otherwise all you requests will be denied. This is explained in the AJAX section of the documentation: CSRF protection in CubicWeb

The whole mechanism is explained in the documentation: CSRF protection in CubicWeb

🚧 Other breaking changes#

We decided to stop releasing cubicweb as debian packages that we used on multi-purpose servers in favor of docker images that we run with docker-compose or on kubernetes. Thanks for all the fishes.

🎉 New features#

  • add a component to disable RQL suggestions: cubicweb.web.views.magicsearch.RQLNoSuggestionsBuilder

👷 Bug fixes#

  • [reledit] display reledit for a relation if some conditions are satisfied ([1] the relation don’t have rqlexpr permissions and can be deleted [2] at least one of related entites can be deleted)

  • pyramid/predicates: avoid to show an error without a session connection

  • be sure db-statement-timeout is not None

  • correctly transform cubicweb.web.RemoteCallFailed into pyramid corresponding exceptions, this allow to propagate the correct content type (for example for json exceptions)

  • “cubicweb-ctl list” now supports multiple dependencies constraints

🤷 Various changes#

  • fix error cases when internationalizable is not defined on rdef

  • improve docstring in web.views.basecontrollers

🤖 Continuous integration#

  • coverage: gitlab-ci is able to read the coverage report we produce

  • disable from-forge for now since we aren’t using them

  • fix path to coverage-*.xml for non-reports artifacts

  • flake8: integrate flake8-gl-codeclimate for QA reports

  • integrate junit reports style for tests errors in gitlab

  • optimisation: allow to interrupt started jobs that can be replaced

  • pytest-html: generate self contained html file for easier test repport browsing

  • trigger py3-* jobs on tox.ini/.gitlab-ci.yml/requirements modifications

  • use gitlab readthedocs integration

📋 Developer experience#

  • using black on the whole project o/ (thx for hg format-source)

  • debug/ux: display traceback of stderr when exception in addition of the html page

  • testing: activate debug mode during testing

  • ux: display on stdout the requests made to the server like nginx

  • ux: display traceback on stderr on request failure

  • ux: logger.info for selected view by ViewController

3.31.9 (2021-11-17)#

👷 Bug fixes#

  • pkg: pin Yams version < 0.46.0

3.31.8 (2021-11-17)#

🤷 Various changes#

  • depend on sphinx >= 4.3

3.31.7 (2021-11-02)#

👷 Bug fixes#

  • build_doc: docutils version 0.18.0 breaks doc building (#443)

  • pkg: don’t use pyparsing 3 since it’s not compatible with rdflib 5. (#441)

  • pkg: pin pyramid_multiauth version to avoid compatibility issue with Pyramid 1 (#450)

3.31.6 (2021-09-28)#

🤷 Various changes#

  • backout “limit setuptools version to avoid issue with 2to3”

  • use our package rdflib-jsonld-without-2to3, this is a fork of rdflib-jsonld with 2to3 usage removed, but which still contains the whole package code unlike rdflib-jsonld 0.6.x.

3.31.5 (2021-09-24)#

👷 Bug fixes#

  • setup: keep rdflib-jsonld at version < 0.6.0

  • setup: limit setuptools version to avoid issue with 2to3

3.31.4 (2021-09-14)#

  • upgrade rdflib-jsonld version to keep compatibility with setupools 58 and above

3.31.3 (2021-07-23)#

👷 Bug fixes#

  • pin rdflib < 6.0.0 to avoid compatibility issues

🤖 Continuous integration#

  • use image from heptapod registry since r.intra was shut down

3.31.2 (2021-07-19)#

👷 Bug fixes#

  • do not consume a lot of time to collect debug data if no one is listening on debug channels

  • fix incomplete merge of previous versions (brings back write_front)

3.31.1 (2021-05-18)#

Revert#

  • backed out changeset bcb633bd791d, don’t give event to notify_on_commit Notification are done using Operation, and Operation do not have a event attribute, because they can be used for several event. Moreover, this commit (bcb633bd791d) changed the prototype of the notify_on_commit without giving the right event to the Operation (which is a singleton).

3.31 (2021-05-04)#

🎉 New features#

  • handle same_site cookies configuration in pyramid.ini

  • order: add support for order by NULLS LAST and NULLS FIRST

  • improve default cubicweb skeleton

👷 Bug fixes#

  • create anonymous user at runtime if it doesn’t exist already.

  • dbcreate: don’t ask confirmation to create schema in automatic

  • hooks/notification: BREAKING CHANGE correctly initialize operation with event attribute

  • RQLExpression: performance issue on RQLExpressions using EXISTS() BREAKING CHANGE: explicitly use EXISTS in RQLExpression for permissions

  • fix some security issues

Documentation#

  • tuto: add structure of “enhance views” museum tutorial part.

  • tuto: redact “React in a CubicWeb view” museum tuto part.

  • tuto: rename cubicweb-tuto to tuto, avoiding confusion with cubicweb_tuto

🤖 Continuous integration#

  • gitlab-ci: set expiration delay to 2 weeks for artifacts

  • image is no longer a global keyword, use default

  • rename jobs names to match global conventions

  • test skeleton own tox in the CI

🤷 Various changes#

  • cleanup: Remove migrations for 3.21 and less

  • py3: Rename raw_input (that does not exist anymore) to input

  • tests: create a .nobackup file in the data/database directory (#298)

3.30.1 (2021-07-23)#

👷 Bug fixes#

  • pin rdflib < 6.0.0 to avoid compatibility issues

🤖 Continuous integration#

  • use image from heptapod registry since r.intra was shut down

3.30.0 (2021-03-16)#

🎉 New features#

  • config: read required variables from environment (#85)

  • db-create:  add drop option to control database deletion (#202) BREAKING CHANGE: cubicweb-ctl db-create no more drops the db in

  • massive store: add an option to allow stores not to drop constraints (#219)

  • pyramid-ctl: add “nb-threads” parameter to cubicweb-ctl pyramid (#119)

  • urlpublish: add empty_rset_raises_404 flag on rql rewrite urls (#199)

  • add script_attribute to add_js function (#210)

  • cubicweb/cwconfig: authenticated SMTP outgoing email

  • database/exception: include the query information in database error for better debuging

  • upgrade Logilab’s dependencies to last versions

  • web: only set “Vary: Accept-Language” when we translate something (#224)

👷 Bug fixes#

  • rql2sql: properly handle date and datetime operations with SQLite (#109)

  • rql: make the rql completion working again

  • rql: refactor GROUP_CONCAT so that it handles NULL values

  • catch authentication exception

  • ci: manually remove the .tox/doc directory (#206)

  • ci: use **/*.py to match all python files

  • ci:  recreate doc environment from scratch (#206)

  • cwgettext: missing local module

  • db-create: don’t force to use –drop if there is no existing db.

  • deprecated: logilab.common.deprecated has been renamed to callable_deprecated

  • deps: we are not yet compatible with pyramid 2.0

  • migractions: don’t use notification hooks during postcreate

  • py3: we still have some unicode() arround

  • repo_cnx: Catch OperationalError during repo_cnx (#215)

  • skeleton: add gitlab-ci in skeleton manifest

  • skeleton: make the skeleton black compliant

  • skip a wdoc test when doctuils is not available

  • typo: drop_db instead of drop_pd

  • urlpublisher: raise a 404 when a URL rewrite with rql has no rset (#199)

  • UX when migractions failed to get its connection

  • migractions: don’t commit in the middle of drop_cube

  • views: fix possible UnboundLocalError in ErrorView

  • server: Set language of connection in all cases (#87)

Documentation#

  • deploy: add a Docker section in deployment

  • deploy: Update kubernetes deploy

  • deploy: Update uwsgi deployment

  • deploy:  add section ref for kubernetes section

  • include api documentation

  • mention weekly meeting in matrix

  • Add link to migration and remove FIXME

  • add more links in the index and capitalize entries (#185)

  • all-in-one.conf: add link in index.rst

  • dataimport: remove SQLGenObjectStore description and add MassiveObjectStore.

  • index: remove “skeleton”, since it’s already explain in “layout”

  • licence: automatically set licence info in setup.py template (#94)

  • move (and fix) apache documentation to the deply section

  • Remove SQLSERVER

  • rql: replace COMMA_JOIN by GROUP_CONCAT (#259)

  • tutorials: correct a dead link.

  • tutorials: add a link to museum demo source code, and correct a typo.

  • tutorials: add introduction and structure of the museum tutorial.

  • tutorials: redaction of “data-management/import” part of the museum tuto.

  • tutorials: redaction of “getting started” part.

  • tutorials: reword

🤖 Continuous integration#

🤷 Various changes#

  • remove statsd (closes #39) BREAKING CHANGE

  • remove web.cors in favor of wsgicors with pyramid

  • server/migractions: simplify the Migration Handler entry point

  • server: replace utils.QueryCache with cachetools.LFUCache

  • Silent yams warning (first rdef selection from an ambiguous rtype)

  • Very minor improvements of cubicweb/server/repository.py

  • views: Make JsonMixIn.wdata method usable with non-web connections

3.29.6 (2021-10-07)#

🎉 New features#

  • Allow authentication on SMTP for outgoing emails. This feature was present in 3.28, but got lost during merge.

3.29#

🎉 New features#

  • we started to reorganize the documentation, in particular the index, this work will continue with the next releases. Thus, some chapters are still missing

  • ext/markdown: add urlize extension to auto link url in markdown documents

👷 Bug fixes#

  • .gitlab-ci.yaml.tmpl should be named .gitlab-ci.yml.tmpl for heptapod

Documentation#

  • Change data model link to use the definition from YAMS

  • reorganize the TOC

  • fix a few dead links

  • add explanation on site_cubicweb.py

  • update cube layout documentation

  • add more explanations to what is a cube

  • pooler: bad option name for the connections pooler

  • Change data model link to use the definition from YAMS

🤖 Continuous integration#

  • build the documentation on gitlab pages

🤷 Various changes#

  • DeprecatedWarning: [logilab.common.deprecation] moved has been renamed and is deprecated, uses callable_moved instead

  • add pipeline badge and shields with stats from pypi & docker

3.28.2#

Fixed#

  • re-introduce cubicweb.pyramid.resources.EntityResource/ETypeResource

  • re-introduce cubicweb.pyramid.predicates.MatchIsETypePredicate

  • both were still needed for several cubes

3.28.1#

Fixed#

  • python 3.8 compatibility: base64 encodestring has been removed, use encodebytes

3.28#

The big highlights of this release are:

  • CubicWeb now requires python >=3.6

  • a new dynamic database connections pooler to replace the old static one

  • a big upgrade in our CI workflow both for tests and documentations

  • RDF generations when rdf mimetype in Accept HTTP headers

  • rql resultset now stores selections variables names for RQL select queries, this will allow to build better tools

Added#

  • [pyramid]`has_cw_permission` pyramid predicates added for routes and view

  • The database pooler is now dynamic. New connections are opened when needed and closed after a configurable period of time. This can be configured through connections-pooler-max-size (default 0, unlimited), connections-pooler-min-size (default 0), and connections-pooler-idle-timeout (default 600 seconds). The old configuration connections-pooler-size has been dropped.

  • [pyramid-debugtoolbar] make SQL and RQL tables sortable

  • [RQL]Resulset now stores selected variables for RQL select queries

  • pyramid: add routes /{eid} and /{etype}/{eid} to return RDF when rdf mimetype in Accept HTTP headers

  • entities: simplify rdf generation and add a generic rdf adapter

  • web.views: add Link alternate in HTTP response header in HTML view

  • Black and Mypy config in tox.ini file in new cube skeleton

  • Gitlab CI config file in new cube skeleton

Changed#

  • CubicWeb now requires python >=3.6

  • CI now test Cubicweb against latest unreleased public commits of its dependencies

  • CI now rebuilds documentation after tests

Deprecated#

  • Class cubicweb.view.EntityAdapter was moved to cubicweb.entity.EntityAdapter, a deprecation warning is in place, but please update your source code accordingly

Removed#

  • Support for plpython has been dropped

  • RDFnquadsView (Breaking Change)

Fixed#

  • Fix various tests in the CI

  • Use SchemaLoader instead of pyfilereader

  • [pyramid-debugtoolbar] remove CW controller panel rendering when no controller got collected

  • [basecontroller] link tags in the header can only be added on entities

  • add a __contains__ method to dict_protocol_catcher to avoid breaking on “in”

Thanks to our contributors: Simon Chabot, Laurent Peuch, Nicolas Chauvat, Philippe Pepiot, Élodie Thieblin, François FERRY, Fabien Amarger, Laurent Wouters, Guillaume Vandevelde.

3.27 (31 January 2020)#

New features#

  • Tests can now be run concurrently across multiple processes. You can use pytest-xdist for that. For tests using PostgresApptestConfiguration you should be aware that startpgcluster() can’t run concurrently. Workaround is to call pytest with --dist=loadfile to use a single test process per test module or use an existing database cluster and set db-host and db-port of devtools.DEFAULT_PSQL_SOURCES['system'] accordingly.

  • on cubicweb-ctl create and cubicweb-ctl pyramid, if it doesn’t already exist in the instance directory, the pyramid.ini file will be generated with the needed secrets.

  • add a –pdb flag to all cubicweb-ctl command to launch (i)pdb if an exception occurs during a command execution.

  • the –loglevel and –dbglevel flags are available for all cubicweb-ctl instance commands (and not only the pyramid one)

  • following “only in foreground” behavior all commands logs to stdout by default from now on. To still log to a file pass log_to_file=True to CubicWebConfiguration.config_for

  • add a new migration function update_bfss_path(old_path, new_path) to update the path in Bytes File-System Storage (bfss).

  • on every request display request path and selected controller in CLI

  • migration interactive mode improvements:

    • when an exception occurs, display the full traceback instead of only the exception

    • on migration p(db) choice, launch ipdb if it’s installed

    • on migration p(db) choice, give the traceback to pdb if it’s available, this mean that the (i)pdb interactive session will be on the stack of the exception instead of being on the stack where pdb is launched which will allow the user to access all the relevant context of the exception which otherwise is lost

  • on DBG_SQL and/or DBG_RQL, if pygments is installed, syntax highlight sql/rql debug output

  • allow to specify the instance id for any instance command using the CW_INSTANCE global variable instead of or giving it as a cli argument

  • when debugmode is activated (‘-D/–debug’ on the pyramid command for example), the HTML generated by CW will contains new tags that will indicate by which object in the code it has been generated and in which line of which source code. For example:

<div
  cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
  cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/web/views/basetemplates.py:161"
  id="contentmain">
    <h1
      cubicweb-generated-by="cubicweb.web.views.basetemplates.TheMainTemplate"
      cubicweb-from-source="/home/user/code/logilab/cubicweb/cubicweb/view.py:136">
        unset title
    </h1>
    [...]
</div>

While this hasn’t been done yet, this feature is an open path for building dynamic tools that can help inspect the page.

  • a new debug channels mechanism has been added, you can subscribe to one of those channels in your python code to build debug tools for example (the pyramid custom panels are built using that) and you will receive a datastructure (a dict) containing related information. The available channels are: controller, rql, sql, vreg, registry_decisions

  • add a new ‘-t/–toolbar’ option the pyramid command to activate the pyramid debugtoolbar

  • a series of pyramid debugtoolbar panels specifically made for CW, see bellow

Pyramid debugtoolbar and custom panel#

The pyramid debugtoolbar is now integrated into CubicWeb during the development phase when you use the ‘pyramid’ command. To activate it you need to pass the ‘-t/–toolbar’ argument to the ‘pyramid’ command.

In addition, a series of custom panels specifically done for CW are now available, they display useful information for the development and the debugging of each page. The available panels are:

  • a general panel which contains the selected controller, the current settings and useful links screenshot

  • a panel listing all decisions taken in registry for building this page screenshot

  • a panel listing the content of the vreg registries screenshot

  • a panel listing all the RQL queries made during a request screenshot

  • a panel listing all the SQL queries made during a request screenshot

Furthermore, in all those panels, next to each object/class/function/method a link to display its source code is available (shown as ‘[source]’ screenshot) and also every file path shown is a traceback is also a link to display the corresponding file (screenshot). For example: screenshot.

Backwards incompatible changes#

  • Standardization on the way to launch a cubicweb instance, from now on the only way to do that will be the used the pyramid command. Therefore:

    • cubicweb-ctl commands “start”, “stop”, “restart”, “reload” and “status” have been removed because they relied on the Twisted web server backend that is no longer maintained nor working with Python 3.

    • Twisted web server support has been removed.

    • cubicweb-ctl wsgi has also been removed.

  • Support for legacy cubes (in the ‘cubes’ python namespace) has been dropped. Use of environment variables CW_CUBES_PATH and CUBES_DIR is removed.

  • Python 2 support has been dropped.

  • Exceptions in notification hooks aren’t catched-all anymore during tests so one can expect tests that seem to pass (but were actually silently failing) to fail now.

  • All “cubicweb-ctl” command only accept one instance argument from now one (instead of 0 to n)

  • ‘pyramid’ command will always run in the foreground now, by consequence the option --no-daemon has been removed.

  • DBG_MS flag has been removed since it is not used anymore

  • transactions db logs where displayed using the logging (debug/info/warning…) mechanism, now it is only displayed if the corresponding DBG_OPS flag is used

  • backward python 2 compatible code for scheduler class has been removed from cubicweb.server.utils. If you get an import error when doing from cubicweb.server.utils import scheduler replace it with from sched import scheduler.

Deprecated code drops#

Most code deprecated until version 3.25 has been dropped.

3.26 (1 February 2018)#

New features#

Backwards incompatible changes#

  • CubicWebConfiguration method ‘shared_dir’ got dropped.

3.25 (14 April 2017)#

New features#

  • A new option connections-pooler-enabled (default yes) has been added. This allow to switch off internal connection pooling for use with others poolers such as pgbouncer.

  • In deleteconf view (confirmation before deletion), the list of first-level composite objects that would be deleted along with the primary entity is displayed (01eeea97e549).

  • The cubicweb.pyramid module now provides a Paste application factory registered as an entry point named pyramid_main and that can be used to run a Pyramid WSGI application bound to a CubicWeb repository.

  • A new configuration type pyramid has been added to create CubicWeb’s instances (through cubicweb-ctl create -c pyramid <basecube> <appid>). This configuration bootstraps a CubicWeb instance that is essentially a repository plus the minimal setup to run a Pyramid WSGI application on top of it. Noticeably, it does not ship all web configuration but rather relies on configuration declared in a development.ini file for any Pyramid application.

  • A new way to declare workflows as simple data structure (dict/list) has been introduced. Respective utility functions live in cubicweb.wfutils module. This handles both the creation and migration of workflows.

  • A new IDublinCore adapter has been introduced to control the generation of Dublin Core metadata that are used in several base views.

  • It is now possible to derive rtags using their derive method (0849a5eb57b8). Derived rtags keep a reference to the original rtag and only hold custom rules, allowing changes which are done in the original rtag after derivation to be still considered.

  • A new cubicweb-ctl scheduler <appid> command has been introduced to run background and periodic tasks of the repository (previously called looping tasks). In a production environment, a process with this command should be run alongside with a WSGI server process (possibly running multiple processes itself).

Backwards incompatible changes#

  • As a consequence of the replacement of the old looping tasks manager by a scheduler, all cubicweb-ctl’s “start” commands (i.e. start, pyramid, wsgi) do not start repository looping tasks manager anymore, nor do they start the scheduler. Site administrators are thus expected to start this scheduler as a separate process. Also, registering looping tasks (i.e. calling repo.looping_tasks()) is a no-op when the repository has no scheduler set; a warning is issued in such cases. Application developers may rely on repository’s has_scheduler method to determine if they should register a looping task or not.

  • In cubicweb.pyramid, function make_cubicweb_application got renamed into config_from_cwconfig (950ce7d9f642).

  • Several cleanups in repository’s session management have been conducted resulting from changes introduced in 3.19 release. Among others, the cubicweb.server.session.Session class has been dropped, and request session attribute is now tight to a web session whose implementation depends on the front-end used (twisted or pyramid). Hence this attribute should not be accessed from “repository side” code (e.g. hooks or operations) and has lost some of his former attributes like repo which used to reference the repository instance. Due to this, you don’t have anymore access to session’s data through the connection, which leds to deprecation of the data attribute and removal of get_shared_data and set_shared_data methods which are deprecated since 3.19.

  • Support for ‘https-url’ configuration option has been removed (4516c3956d46).

  • The next_tabindex method of request class has been removed (011730a4af73). This include the removal of settabindex from the FieldWidget class init method.

  • The cubicweb.hook.logstats.start hook was dropped because it’s looping task would not be run in a web instance (see first point about repository scheduler).

  • uicfg rules to hide the opposite relation of inlined form are not anymore automatically added, because this was actually done randomly and so not reliable, so you’ll have to add them manually:

    autoform_section.tag_subject_of(('CWUser', 'use_email', 'EmailAddress'),
                                'main', 'inlined')
autoform_section.tag_object_of(('CWUser', 'use_email', 'EmailAddress'),
                               'inlined', 'hidden')

3.24 (2 November 2016)#

New features#

  • Various bits of a CubicWeb application configuration can be now be overridden through environments variables matching configuration option names prefixed by CW_ (for instance CW_BASE_URL).

  • Cubes are now standard Python packages named as cubicweb_<cubename>. They are not anymore installed in <prefix>/share/cubicweb/cubes. Their discovery by CubicWeb is handled by a new setuptools entry point cubicweb.cubes. A backward compatibility layer is kept for “legacy” cubes.

  • Pyramid support made it into CubicWeb core. Applications that use it should now declare the cubicweb[pyramid] dependency instead of cubicweb-pyramid.

  • New NullStore class in cubicweb.dataimport.stores as new base class for every store, and allowing to test your dataimport chain without actually importing anything.

Major changes#

There has been several important changes to the core internals of CubicWeb:

  • Dropped asource and extid columns from the entities table as well as the index on the type column, for a sensible optimization on both massive data insertion and database size / index rebuilding.

  • Dropped the moved_entities table and related mecanism to remember that an entity has been moved from a source to the system database - this is now the responsability of source’s parser to detect this (usually by remembering its original external id as cwuri).

  • Dropped the original ‘give me an eid for this extid, but call me back on another line if it has to be created’ mecanism on which the cwxmlparser was relying, in favor of parsers using the dataimport API. This includes dropping the cwxmlparser. If you’re using it, you’ll have to write a specific parser, examples to come.

  • Dropped source mapping handling (schema, views, logic) with its client the cwxmlparser. This is not worth the burden, specific parsers should be preferred.

The above changes lead to the following API changes:

  • req.entity_metas(eid) doesn’t return anymore a ‘type’ nor ‘source’ keys, use req.entity_type(eid) instead or ‘cw_source’ relation to get those information,

  • deprecated entity.cw_metainformation(), which doesn’t return anymore it’s ‘source’ key,

  • dropped repository.type_and_source_from_eid(eid, cnx), repository.extid2eid(…) and source.eid_type_source(cnx, eid),

  • dropped source.support_entity(etype) and source.support_relation(rtype),

  • dropped ‘cw_source’ key from default JSON representation of an entity,

  • dropped source_uris() and handle_deletion(…) method from datafeed parser base class, deletion of entities is now the responsability of specific implementation (see ldapparser for example),

  • entities from external source are always displayed in the UI with a link to the local entity, not the original one simplifying entity.absolute_url() implementation and allowing to drop use_ext_eid argument of entity.rest_path() (though it’s still supported for backward compat).

Changes to the massive store#

Several improvments have been done to cubicweb.dataimport.massive_store, with among the more important ones:

  • Extended store API to provide more control to end-users: fill_entities_table, fill_relation_table, fill_meta_relation_table.

  • Dropped on_commit / on_rollback arguments of the constructor.

  • Use a slave specific temporary table for entities insertion as for relations (should improve concurrency when using in master/slaves mode).

  • Delay dropping of constraint to the finish method, avoiding performance problem that was occuring because indexes were dropped at store creation time.

  • Consider the given metadata generator when looking for which metadata tables should have their constraints dropped.

  • Don’t drop index on entities.eid, it’s too costly to rebuild on database with some million of entities.

3.23 (24 June 2016)#

New features#

  • Python 3.x support in CubicWeb itself is now complete, except for the twisted package (since Twisted does not completely support Python 3.x itself). The skeleton for new cube should also be Python 3 compatible, in particular its setup.py got updated.

  • The source-sync command can now synchronize all sources in the database, if no <source> argument is provided.

  • Datafeed source synchronization is now asynchronous when requested from user interface.

  • Most indexes and constraints will be rebuilt during the migration, because they are now named after a md5 hash to control the name’s size.

  • Index are renamed upon renaming of an entity type, so they are still correctly tracked.

  • A new db-check-index command is added to cubicweb-ctl, to display the differences between the indexes in the database and those expected by the schema. It’s recommended to run this command after the migration to 3.23 and to adjust things manually for cases that are not easily handled by the migration script, such as indexes of entity types that have been renamed. It should be mostly about dropping extra indexes.

  • Deprecated MetaGenerator in favor of slightly adapted API in MetadataGenerator (more consistent, giving more control to sub-classes and suitable for usage with the MassiveObjectStore)

  • Major cleanups of the MassiveObjectStore and its PGHelper companion class:

    • dropped a bunch of unnecessary / unused attributes

    • refactored / renamed internal methods

    • added support for a metadata generator, the now recommended way to control metadata generation

  • Deprecated SQLGenObjectStore, MassiveObjectStore should be used instead.

Backwards-incompatible changes#

  • Generative tests à la logilab-common are not supported anymore in CubicWebTC. It is advised to use the subtests API (available on CubicWebTC either from the standard library as of Python 3.4 or through unittest2 package otherwise).

  • CubicWebTC’s set_description method (comming from logilab.common.testlib.TestCase) is no longer available.

Development#

When installed within a virtualenv, CubicWeb will look for instances data as in user mode by default, that is in $HOME/etc/cubicweb.d, as opposed to $VIRTUAL_ENV/etc/cubicweb.d previously. To restore this behavior, explicitly set CW_MODE to system. Alternatively (and preferably), the CW_INSTANCES_DIR environment variables may be used to specify instances data location.

3.22 (4 January 2016)#

New features#

  • a huge amount of changes were done towards python 3.x support (as yet incomplete). This introduces a new dependency on six, to handle python2/python3 compatibility.

  • new cubicweb.dataimport.massive_store module, a postgresql-specific store using the COPY statement to accelerate massive data imports. This functionality was previously part of cubicweb-dataio (there are some API differences with that previous version, however).

  • cubes custom sql scripts are executed before creating tables. This allows them to create new types or extensions.

  • the ejsonexport view can be specialized using the new ISerializable entity adapter. By default, it will return an entity’s (non-Bytes and non-Password) attributes plus the special cw_etype and cw_source keys.

  • cubes that define custom final types are now handled by the add_cube migration command.

  • synchronization of external sources can be triggered from the web interface by suitably privileged users with a new cw.source-sync action.

User-visible changes#

  • the ldapfeed source now depends on the ldap3 module instead of python-ldap.

  • replies don’t get an Expires header by default. However when they do, they also get a coherent Cache-Control.

  • data files are regenerated at each request, they are no longer cached by cubicweb.web.PropertySheet. Requests for data files missing the instance hash are handled with a redirection instead of a direct reply, to allow correct cache-related reply headers.

API changes#

  • config.repository() creates a new Repository object each time, instead of returning a cached object. WARNING: this may cause unexpected issues if several repositories end up being used.

  • migration scripts, as well as other scripts executed by cubicweb-ctl shell, are loaded with the print_function flag enabled (for backwards compatibility, if that fails they are re-loaded without that flag)

  • the cw_fti_index_rql_queries method on entity classes is replaced by cw_fti_index_rql_limit, a generator which yields ResultSet objects containing entities to be indexed. By default, entities are returned 1000 at a time.

  • IDownloadableAdapter API is clarified: download_url, download_content_type and download_file_name return unicode objects, download_data returns bytes.

  • the Repository.extid2eid() entry point for external sources is deprecated. Imports should use one of the stores from the cubicweb.dataimport package instead.

  • the cubicweb.repoapi.get_repository() function’s uri argument should no longer be used.

  • the generic datafeed xml parser is deprecated in favor of the “store” API introduced in cubicweb 3.21.

  • the session manager lives in the sessions registry instead of components.

  • TZDatetime attributes are returned as timezone-aware python datetime objects. WARNING: this will break client applications that compare or use arithmetic involving timezone-naive datetime objects.

  • creation_date and modification_date attributes for all entities are now timezone-aware (TZDatetime) instead of localtime (Datetime). More generally, the Datetime type should be considered as deprecated.

Deprecated code drops#

  • the cubicweb.server.hooksmanager module was removed

  • the Repository.pinfo() method was removed

  • the cubicweb.utils.SizeConstrainedList class was removed

  • the ‘startorder’ file in configuration directory is no longer honored

3.21 (10 July 2015)#

New features#

  • the datadir-url configuration option lets one choose where static data files are served (instead of the default ${base-url}/data/)

  • some integrity checking that was previously implemented in Python was moved to the SQL backend. This includes some constraints, and referential integrity. Some consequences are that:

    • disabling integrity hooks no longer disables those checks

    • upgrades that modify constraints will fail when running on sqlite (but upgrades aren’t supported on sqlite anyway)

    Note: as of 3.21.0, the upgrade script only works on PostgreSQL. The migration for SQLServer will be added in a future bugfix release.

  • for easier instance monitoring, cubicweb can regularly dump some statistics (basically those exposed by the ‘info’ and ‘gc’ views) in json format to a file

User-visible changes#

  • the use of fckeditor for text form fields is disabled by default, to re-enable it simply install the cubicweb-ckeditor cube (then add_cude(‘ckeditor’) in a migration or in the shell)

  • the ‘https-deny-anonymous’ configuration setting no longer exists

Code movement#

The cubicweb.web.views.timeline module (providing the timeline-json, timeline and static-timeline views) has moved to a standalone cube

API changes#

  • req.set_cookie’s “expires” argument, if not None, is expected to be a date or a datetime in UTC. It was previously interpreted as localtime with the UTC offset the server started in, which was inconsistent (we are not aware of any users of that API).

  • the way to run tests on a postgresql backend has changed slightly, use cubicweb.devtools.{start,stop}pgcluster in setUpModule and tearDownModule

  • the Connection and ClientConnection objects introduced in CubicWeb 3.19 have been unified. To connect to a repository, use:

    session = repo.new_session(login, password=...)
with session.new_cnx() as cnx:
    cnx.execute(...)

    In tests, the ‘repo_cnx’ and ‘client_cnx’ methods of RepoAccess are now aliases to ‘cnx’.

Deprecated code drops#

  • the user_callback api has been removed; people should use plain ajax functions instead

  • the Pyro and Zmq-pickle remote repository access methods have been entirely removed (emerging alternatives such as rqlcontroller and cwclientlib should be used instead). Note that as a side effect, “repository-only” instances (i.e. without a http component) are no longer possible. If you have any such instances, you will need to rename the configuration file from repository.conf to all-in-one.conf and run cubicweb-ctl upgrade to update it. Likewise, remote cubicweb-ctl shell is no longer available.

  • the old (deprecated since 3.19) DBAPI api is completely removed

  • cubicweb.toolsutils.config_connect() has been removed

3.20 (06/01/2015)#

New features#

  • virtual relations: a new ComputedRelation class can be used in schema.py; its rule attribute is an RQL snippet that defines the new relation.

  • computed attributes: an attribute can now be defined with a formula argument (also an RQL snippet); it will be read-only, and updated automatically.

    Both of these features are described in CWEP-002, and the updated “Data model” chapter of the CubicWeb book.

  • cubicweb-ctl plugins can use the cubicweb.utils.admincnx function to get a Connection object from an instance name.

  • new ‘tornado’ wsgi backend

  • session cookies have the HttpOnly flag, so they’re no longer exposed to javascript

  • rich text fields can be formatted as markdown

  • the edit controller detects concurrent editions, and raises a ValidationError if an entity was modified between form generation and submission

  • cubicweb can use a postgresql “schema” (namespace) for its tables

  • “cubicweb-ctl configure” can be used to set values of the admin user credentials in the sources configuration file

  • in debug mode, setting the _cwtracehtml parameter on a request allows tracing where each bit of output is produced

API Changes#

  • ucsvreader() and ucsvreader_pb() from the dataimport module have 2 new keyword arguments delimiter and quotechar to replace the separator and quote arguments respectively. This makes the API match that of Python’s csv.reader(). The old arguments are still supported though deprecated.

  • the migration environment’s remove_cube function is now called drop_cube.

  • cubicweb.old.css is now cubicweb.css. The previous “new” cubicweb.css, along with its cubicweb.reset.css companion, have been removed.

  • the jquery-treeview plugin was updated to its latest version

Deprecated Code Drops#

  • most of 3.10 and 3.11 backward compat is gone; this includes:

    • CtxComponent.box_action() and CtxComponent.build_link()

    • cubicweb.devtools.htmlparser.XMLDemotingValidator

    • various methods and properties on Entities, replaced by cw_edited and cw_attr_cache

    • ‘commit_event’ method on hooks, replaced by ‘postcommit_event’

    • server.hook.set_operation(), replaced by Operation.get_instance(…).add_data()

    • View.div_id(), View.div_class() and View.create_url()

    • *VComponent classes

    • in forms, Field.value() and Field.help() must take the form and the field itself as arguments

    • form.render() must get w as a named argument, and renderer.render() must take w as first argument

    • in breadcrumbs, the optional recurs argument must be a set, not False

    • cubicweb.web.views.idownloadable.{download_box,IDownloadableLineView}

    • primary views no longer have render_entity_summary and summary methods

    • WFHistoryVComponent’s cell_call method is replaced by render_body

    • cubicweb.dataimport.ObjectStore.add(), replaced by create_entity

    • ManageView.{folders,display_folders}

3.19 (28/04/2015)#

New functionalities#

  • implement Cross Origin Resource Sharing (CORS) (see #2491768)

  • system_source.create_eid can get a range of IDs, to reduce overhead of batch entity creation

Behaviour Changes#

  • The anonymous property of Session and Connection are now computed from the related user login. If it matches the anonymous-user in the config the connection is anonymous. Beware that the anonymous-user config is web specific. Therefore, no session may be anonymous in a repository only setup.

New Repository Access API#

A new explicit Connection object replaces Session as the main repository entry point. Connection holds all the necessary methods to be used server-side (execute, commit, rollback, call_service, entity_from_eid, etc…). One obtains a new Connection object using session.new_cnx(). Connection objects need to have an explicit begin and end. Use them as a context manager to never miss an end:

with session.new_cnx() as cnx:
    cnx.execute('INSERT Elephant E, E name "Babar"')
    cnx.commit()
    cnx.execute('INSERT Elephant E, E name "Celeste"')
    cnx.commit()
# Once you get out of the "with" clause, the connection is closed.

Using the same Connection object in multiple threads will give you access to the same Transaction. However, Connection objects are not thread safe (hence at your own risks).

repository.internal_session is deprecated in favor of repository.internal_cnx. Note that internal connections are now safe by default, i.e. the integrity hooks are enabled.

Backward compatibility is preserved on Session.

A new API has been introduced to replace the dbapi. It is called repoapi.

There are three relevant functions for now:

  • repoapi.get_repository returns a Repository object either from an URI when used as repoapi.get_repository(uri) or from a config when used as repoapi.get_repository(config=config).

  • repoapi.connect(repo, login, **credentials) returns a ClientConnection associated with the user identified by the credentials. The ClientConnection is associated with its own Session that is closed when the ClientConnection is closed. A ClientConnection is a Connection-like object to be used client side.

  • repoapi.anonymous_cnx(repo) returns a ClientConnection associated with the anonymous user if described in the config.

On the client/web side, the Request is now using a repoapi.ClientConnection instead of a dbapi.connection. The ClientConnection has multiple backward compatible methods to make it look like a dbapi.Cursor and dbapi.Connection.

Session used on the Web side are now the same than the one used Server side. Some backward compatibility methods have been installed on the server side Session to ease the transition.

The authentication stack has been altered to use the repoapi instead of the dbapi. Cubes adding new element to this stack are likely to break.

Session data can be accessed using the cnx.data dictionary, while transaction data is available through cnx.transaction_data. These replace the [gs]et_shared_data methods with optional txid kwarg.

All current methods and attributes used to access the repo on CubicWebTC are deprecated. You may now use a RepoAccess object. A RepoAccess object is linked to a new Session for a specified user. It is able to create Connection, ClientConnection and web side requests linked to this session:

access = self.new_access('babar') # create a new RepoAccess for user babar
with access.repo_cnx() as cnx:
    # some work with server side cnx
    cnx.execute(...)
    cnx.commit()
    cnx.execute(...)
    cnx.commit()

with access.client_cnx() as cnx:
    # some work with client side cnx
    cnx.execute(...)
    cnx.commit()

with access.web_request(elephant='babar') as req:
    # some work with client side cnx
    elephant_name = req.form['elephant']
    req.execute(...)
    req.cnx.commit()

By default testcase.admin_access contains a RepoAccess object for the default admin session.

API changes#

  • RepositorySessionManager.postlogin is now called with two arguments, request and session. And this now happens before the session is linked to the request.

  • SessionManager and AuthenticationManager now take a repo object at initialization time instead of a vreg.

  • The async argument of _cw.call_service has been dropped. All calls are now synchronous. The zmq notification bus looks like a good replacement for most async use cases.

  • repo.stats() is now deprecated. The same information is available through a service (_cw.call_service('repo_stats')).

  • repo.gc_stats() is now deprecated. The same information is available through a service (_cw.call_service('repo_gc_stats')).

  • repo.register_user() is now deprecated. The functionality is now available through a service (_cw.call_service('register_user')).

  • request.set_session no longer takes an optional user argument.

  • CubicwebTC does not have repo and cnx as class attributes anymore. They are standard instance attributes. set_cnx and _init_repo class methods become instance methods.

  • set_cnxset and free_cnxset are deprecated. cnxset are now automatically managed.

  • The implementation of cascading deletion when deleting composite entities has changed. There comes a semantic change: merely deleting a composite relation does not entail any more the deletion of the component side of the relation.

  • _cw.user_callback and _cw.user_rql_callback are deprecated. Users are encouraged to write an actual controller (e.g. using ajaxfunc) instead of storing a closure in the session data.

  • A new entity.cw_linkable_rql method provides the rql to fetch all entities that are already or may be related to the current entity using the given relation.

Deprecated Code Drops#

  • session.hijack_user mechanism has been dropped.

  • EtypeRestrictionComponent has been removed, its functionality has been replaced by facets a while ago.

  • the old multi-source support has been removed. Only copy-based sources remain, such as datafeed or ldapfeed.

3.18 (10/01/2014)#

The migration script does not handle sqlite nor mysql instances.

New functionalities#

  • add a security debugging tool (see #2920304)

  • introduce an add permission on attributes, to be interpreted at entity creation time only and allow the implementation of complex update rules that don’t block entity creation (before that the update attribute permission was interpreted at entity creation and update time)

  • the primary view display controller (uicfg) now has a set_fields_order method similar to the one available for forms

  • new method ResultSet.one(col=0) to retrive a single entity and enforce the result has only one row (see #3352314)

  • new method RequestSessionBase.find to look for entities (see #3361290)

  • the embedded jQuery copy has been updated to version 1.10.2, and jQuery UI to version 1.10.3.

  • initial support for wsgi for the debug mode, available through the new wsgi cubicweb-ctl command, which can use either python’s builtin wsgi server or the werkzeug module if present.

  • a rql-table directive is now available in ReST fields

  • cubicweb-ctl upgrade can now generate the static data resource directory directly, without a manual call to gen-static-datadir.

API changes#

  • not really an API change, but the entity permission checks are now systematically deferred to an operation, instead of a) trying in a hook and b) if it failed, retrying later in an operation

  • The default value storage for attributes is no longer String, but Bytes. This opens the road to storing arbitrary python objects, e.g. numpy arrays, and fixes a bug where default values whose truth value was False were not properly migrated.

  • symmetric relations are no more handled by an rql rewrite but are now handled with hooks (from the activeintegrity category); this may have some consequences for applications that do low-level database manipulations or at times disable (some) hooks.

  • unique together constraints (multi-columns unicity constraints) get a name attribute that maps the CubicWeb contraint entities to corresponding backend index.

  • BreadCrumbEntityVComponent’s open_breadcrumbs method now includes the first breadcrumbs separator

  • entities can be compared for equality and hashed

  • the on_fire_transition predicate accepts a sequence of possible transition names

  • the GROUP_CONCAT rql aggregate function no longer repeats duplicate values, on the sqlite and postgresql backends

Deprecation#

  • pyrorql sources have been deprecated. Multisource will be fully dropped in the next version. If you are still using pyrorql, switch to datafeed NOW!

  • the old multi-source system

  • find_one_entity and find_entities in favor of find (see #3361290)

  • the TmpFileViewMixin and TmpPngView classes (see #3400448)

Deprecated Code Drops#

  • ldapuser have been dropped; use ldapfeed now (see #2936496)

  • action GotRhythm was removed, make sure you do not import it in your cubes (even to unregister it) (see #3093362)

  • all 3.8 backward compat is gone

  • all 3.9 backward compat (including the javascript side) is gone

  • the twisted (web-only) instance type has been removed

3.17 (02/05/2013)#

New functionalities#

  • add a command to compare db schema and file system schema (see #464991)

  • Add CubicWebRequestBase.content with the content of the HTTP request (see #2742453) (see #2742453)

  • Add directive bookmark to ReST rendering (see #2545595)

  • Allow user defined final type (see #124342)

API changes#

  • drop typed_eid() in favour of int() (see #2742462)

  • The SIOC views and adapters have been removed from CubicWeb and moved to the sioc cube.

  • The web page embedding views and adapters have been removed from CubicWeb and moved to the embed cube.

  • The email sending views and controllers have been removed from CubicWeb and moved to the massmailing cube.

  • RenderAndSendNotificationView is deprecated in favor of ActualNotificationOp the new operation use the more efficient data idiom.

  • Looping task can now have a interval <= 0. Negative interval disable the looping task entirely.

  • We now serve html instead of xhtml. (see #2065651)

Deprecation#

  • ldapuser have been deprecated. It’ll be fully dropped in the next version. If you are still using ldapuser switch to ldapfeed NOW!

  • hijack_user have been deprecated. It will be dropped soon.

Deprecated Code Drops#

  • The progress views and adapters have been removed from CubicWeb. These classes were deprecated since 3.14.0. They are still available in the iprogress cube.

  • API deprecated since 3.7 have been dropped.

3.16 (25/01/2013)#

New functionalities#

  • Add a new dataimport store (SQLGenObjectStore). This store enables a fast import of data (entity creation, link creation) in CubicWeb, by directly flushing information in SQL. This may only be used with PostgreSQL, as it requires the ‘COPY FROM’ command.

API changes#

  • Orm: set_attributes and set_relations are unified (and deprecated) in favor of cw_set that works in all cases.

  • db-api/configuration: all the external repository connection information is now in an URL (see #2521848), allowing to drop specific options of pyro nameserver host, group, etc and fix broken ZMQ source. Configuration related changes:

    • Dropped ‘pyro-ns-host’, ‘pyro-instance-id’, ‘pyro-ns-group’ from the client side configuration, in favor of ‘repository-uri’. NO MIGRATION IS DONE, supposing there is no web-only configuration in the wild.

    • Stop discovering the connection method through repo_method class attribute of the configuration, varying according to the configuration class. This is a first step on the way to a simpler configuration handling.

    DB-API related changes:

    • Stop indicating the connection method using ConnectionProperties.

    • Drop _cnxtype attribute from Connection and cnxtype from Session. The former is replaced by a is_repo_in_memory property and the later is totaly useless.

    • Turn repo_connect into _repo_connect to mark it as a private function.

    • Deprecate in_memory_cnx which becomes useless, use _repo_connect instead if necessary.

  • the “tcp://” uri scheme used for ZMQ communications (in a way reminiscent of Pyro) is now named “zmqpickle-tcp://”, so as to make room for future zmq-based lightweight communications (without python objects pickling).

  • Request.base_url gets a secure=True optional parameter that yields an https url if possible, allowing hook-generated content to send secure urls (e.g. when sending mail notifications)

  • Dataimport ucsvreader gets a new boolean ignore_errors parameter.

Unintrusive API changes#

  • Drop of cubicweb.web.uicfg.AutoformSectionRelationTags.bw_tag_map, deprecated since 3.6.

User interface changes#

  • The RQL search bar has now some auto-completion support. It means relation types or entity types can be suggested while typing. It is an awesome improvement over the current behaviour !

  • The action box associated with table views (from tableview.py) has been transformed into a nice-looking series of small tabs; it means that the possible actions are immediately visible and need not be discovered by clicking on an almost invisible icon on the upper right.

  • The uicfg module has moved to web/views/ and ui configuration objects are now selectable. This will reduce the amount of subclassing and whole methods replacement usually needed to customize the ui behaviour in many cases.

  • Remove changelog view, as neither cubicweb nor known cubes/applications were properly feeding related files.

Other changes#

  • ‘pyrorql’ sources will be automatically updated to use an URL to locate the source rather than configuration option. ‘zmqrql’ sources were broken before this change, so no upgrade is needed…

  • Debugging filters for Hooks and Operations have been added.

  • Some cubicweb-ctl commands used to show the output of msgcat and msgfmt; they don’t anymore.

3.15 (12/04/2012)#

New functionnalities#

  • Add Zmq server, based on the cutting edge ZMQ (http://www.zeromq.org/) socket library. This allows to access distant instance, in a similar way as Pyro.

  • Publish/subscribe mechanism using ZMQ for communication among cubicweb instances. The new zmq-address-sub and zmq-address-pub configuration variables define where this communication occurs. As of this release this mechanism is used for entity cache invalidation.

  • Improved WSGI support. While there is still some caveats, most of the code which was twisted only is now generic and allows related functionalities to work with a WSGI front-end.

  • Full undo/transaction support : undo of modification has eventually been implemented, and the configuration simplified (basically you activate it or not on an instance basis).

  • Controlling HTTP status code used is not much more easier :

    • WebRequest now has a status_out attribut to control the response status ;

    • most web-side exceptions take an optional status argument.

API changes#

  • The base registry implementation has been moved to a new logilab.common.registry module (see #1916014). This includes code from :

    • cubicweb.vreg (the whole things that was in there)

    • cw.appobject (base selectors and all).

    In the process, some renaming was done:

    • the top level registry is now RegistryStore (was VRegistry), but that should not impact cubicweb client code ;

    • former selectors functions are now known as “predicate”, though you still use predicates to build an object’selector ;

    • for consistency, the objectify_selector decoraror has hence be renamed to objectify_predicate ;

    • on the CubicWeb side, the selectors module has been renamed to predicates.

    Debugging refactoring dropped the more need for the lltrace decorator. There should be full backward compat with proper deprecation warnings. Notice the yes predicate and objectify_predicate decorator, as well as the traced_selection function should now be imported from the logilab.common.registry module.

  • All login forms are now submitted to <app_root>/login. Redirection to requested page is now handled by the login controller (it was previously handle by the session manager).

  • Publisher.publish has been renamed to Publisher.handle_request. This method now contains generic version of logic previously handled by Twisted. Controller.publish is not affected.

Unintrusive API changes#

  • New ‘ldapfeed’ source type, designed to replace ‘ldapuser’ source with data-feed (i.e. copy based) source ideas.

  • New ‘zmqrql’ source type, similar to ‘pyrorql’ but using ømq instead of Pyro.

  • A new registry called services has appeared, where you can register server-side cubicweb.server.Service child classes. Their call method can be invoked from a web-side AppObject instance using new self._cw.call_service method or a server-side one using self.session.call_service. This is a new way to call server-side methods, much cleaner than monkey patching the Repository class, which becomes a deprecated way to perform similar tasks.

  • a new ajax-func registry now hosts all remote functions (i.e. functions callable through the asyncRemoteExec JS api). A convenience ajaxfunc decorator will let you expose your python function easily without all the appobject standard boilerplate. Backward compatibility is preserved.

  • the ‘json’ controller is now deprecated in favor of the ‘ajax’ one.

  • WebRequest.build_url can now take a __secure__ argument. When True cubicweb try to generate an https url.

User interface changes#

A new ‘undohistory’ view expose the undoable transactions and give access to undo some of them.

3.14 (09/11/2011)#

First notice CW 3.14 depends on yams 0.34 (which is incompatible with prior cubicweb releases regarding instance re-creation).

API changes#

  • Entity.fetch_rql restriction argument has been deprecated and should be replaced with a call to the new Entity.fetch_rqlst method, get the returned value (a rql Select node) and use the RQL syntax tree API to include the above-mentionned restrictions.

    Backward compat is kept with proper warning.

  • Entity.fetch_order and Entity.fetch_unrelated_order class methods have been replaced by Entity.cw_fetch_order and Entity.cw_fetch_unrelated_order with a different prototype:

    • instead of taking (attr, var) as two string argument, they now take (select, attr, var) where select is the rql syntax tree beinx constructed and var the variable node.

    • instead of returning some string to be inserted in the ORDERBY clause, it has to modify the syntax tree

    Backward compat is kept with proper warning, BESIDE cases below:

    • custom order method return something else the a variable name with or without the sorting order (e.g. cases where you sort on the value of a registered procedure as it was done in the tracker for instance). In such case, an error is logged telling that this sorting is ignored until API upgrade.

    • client code use direct access to one of those methods on an entity (no code known to do that).

  • Entity._rest_attr_info class method has been renamed to Entity.cw_rest_attr_info

    No backward compat yet since this is a protected method an no code is known to use it outside cubicweb itself.

  • AnyEntity.linked_to has been removed as part of a refactoring of this functionality (link a entity to another one at creation step). It was replaced by a EntityFieldsForm.linked_to property.

    In the same refactoring, cubicweb.web.formfield.relvoc_linkedto, cubicweb.web.formfield.relvoc_init and cubicweb.web.formfield.relvoc_unrelated were removed and replaced by RelationField methods with the same names, that take a form as a parameter.

    No backward compatibility yet. It’s still time to cry for it. Cubes known to be affected: tracker, vcsfile, vcreview.

  • CWPermission entity type and its associated require_permission relation type (abstract) and require_group relation definitions have been moved to a new localperms cube. With this have gone some functions from the cubicweb.schemas package as well as some views. This makes cubicweb itself smaller while you get all the local permissions stuff into a single, documented, place.

    Backward compat is kept for existing instances, though you should have installed the localperms cubes. A proper error should be displayed when trying to migrate to 3.14 an instance the use CWPermission without the new cube installed. For new instances / test, you should add a dependancy on the new cube in cubes using this feature, along with a dependancy on cubicweb >= 3.14.

  • jQuery has been updated to 1.6.4 and jquery-tablesorter to 2.0.5. No backward compat issue known.

  • Table views refactoring : new RsetTableView and EntityTableView, as well as rewritten an enhanced version of PyValTableView on the same bases, with logic moved to some column renderers and a layout. Those should be well documented and deprecates former TableView, EntityAttributesTableView and CellView, which are however kept for backward compat, with some warnings that may not be very clear unfortunatly (you may see your own table view subclass name here, which doesn’t make the problem that clear). Notice that _cw.view(‘table’, rset, *kwargs) will be routed to the new RsetTableView or to the old TableView depending on given extra arguments. See #1986413.

  • display_name don’t call .lower() anymore. This may leads to changes in your user interface. Different msgid for upper/lower cases version of entity type names, as this is the only proper way to handle this with some languages.

  • IEditControlAdapter has been deprecated in favor of EditController overloading, which was made easier by adding dedicated selectors called match_edited_type and match_form_id.

  • Pre 3.6 API backward compat has been dropped, though data migration compatibility has been kept. You may have to fix errors due to old API usage for your instance before to be able to run migration, but then you should be able to upgrade even a pre 3.6 database.

  • Deprecated cubicweb.web.views.iprogress in favor of new iprogress cube.

  • Deprecated cubicweb.web.views.flot in favor of new jqplot cube.

Unintrusive API changes#

  • Refactored properties forms (eg user preferences and site wide properties) as well as pagination components to ease overridding.

  • New cubicweb.web.uihelper module with high-level helpers for uicfg.

  • New anonymized_request decorator to temporary run stuff as an anonymous user, whatever the currently logged in user.

  • New ‘verbatimattr’ attribute view.

  • New facet and form widget for Integer used to store binary mask.

  • New js_href function to generated proper javascript href.

  • match_kwargs and match_form_params selectors both accept a new once_is_enough argument.

  • printable_value is now a method of request, and may be given dict of

    formatters to use.

  • [Rset]TableView allows to set None in ‘headers’, meaning the label should be fetched from the result set as done by default.

  • Field vocabulary computation on entity creation now takes __linkto information into accounet.

  • Started a cubicweb.pylintext pylint plugin to help pylint analyzing cubes.

RQL#

  • Support for HAVING in ‘SET’ and ‘DELETE’ queries.

  • new AT_TZ function to get back a timestamp at a given time-zone.

  • new WEEKDAY date extraction function

User interface changes#

  • Datafeed source now present an history of the latest import’s log, including global status and debug/info/warning/error messages issued during imports. Import logs older than a configurable amount of time are automatically deleted.

  • Breadcrumbs component is properly kept when creating an entity with ‘__linkto’.

  • users and groups management now really lead to that (i.e. includes groups management).

  • New ‘jsonp’ controller with ‘jsonexport’ and ‘ejsonexport’ views.

Configuration#

  • Added option ‘resources-concat’ to make javascript/css files concatenation optional.

API#

cubicweb_web#

CubicWeb web client core. You’ll need a apache-modpython or twisted publisher to get a full CubicWeb web application

Exceptions#

Base exceptions#

Repository exceptions#

Security Exceptions#

exception cubicweb_web.Unauthorized[source]#

Bases: cubicweb._exceptions.SecurityError

raised when a user tries to perform an action without sufficient credentials

exception cubicweb_web.Forbidden[source]#

Bases: cubicweb._exceptions.SecurityError

raised when a user tries to perform a forbidden action

Source exceptions#

Registry exceptions#

Query exceptions#

Misc#

exception cubicweb_web.ValidationError(entity, errors: Dict, msgargs: Optional[Dict] = None, i18nvalues: Optional[List] = None)[source]#

Bases: yams._exceptions.SchemaError

Validation error details the reason(s) why the validation failed.

Arguments are:

  • entity: the entity that could not be validated; actual type depends on the client library

  • errors: errors dictionary, None key used for global error, other keys should be attribute/relation of the entity, qualified as subject/object using yams.role_name(). Values are the message associated to the keys, and may include interpolation string starting with ‘%(KEY-’ where ‘KEY’ will be replaced by the associated key once the message has been translated. This allows predictable/translatable message and avoid args conflict if used for several keys.

  • msgargs: dictionary of substitutions to be inserted in error messages once translated (only if msgargs is given)

  • i18nvalues: list of keys in msgargs whose value should be translated

Translation will be done in-place by calling translate().

Utilities#

cubicweb.dataimport#

Package containing various utilities to import data into cubicweb.

Utilities#

cubicweb.dataimport.count_lines(*args, **kwargs)#
cubicweb.dataimport.ucsvreader_pb(*args, **kwargs)#
cubicweb.dataimport.ucsvreader(*args, **kwargs)#
cubicweb.dataimport.callfunc_every(func, number, iterable)[source]#

yield items of iterable one by one and call function func every number iterations. Always call function func at the end.

Object Stores#

cubicweb.dataimport.RQLObjectStore#

alias of logilab.common.deprecation.callable_moved.<locals>.callnew

cubicweb.dataimport.NoHookRQLObjectStore#

alias of logilab.common.deprecation.callable_moved.<locals>.callnew

cubicweb_web.predicates#

class cubicweb_web.predicates.paginated_rset(nbpages=1)[source]#

Return 1 or more for result set with more rows than one or more page size. You can specify expected number of pages to the initializer (default to one), and you’ll get that number of pages as score if the result set is big enough.

Page size is searched in (respecting order): * a page_size argument * a page_size form parameters * the navigation.page-size property (see Configuring persistent properties)

cubicweb.pyramid#

cubicweb.pyramid.auth#

CubicWeb AuthTkt authentication policy#

When using the cubicweb.pyramid.auth module, which is the default in most cases, you may have to configure the behaviour of these authentication policies using standard’s Pyramid configuration. You may want to configure in your pyramid configuration file:

Session Authentication

This is a AuthTktAuthenticationPolicy so you may overwrite default configuration values by adding configuration entries using the prefix cubicweb.auth.authtkt.session. Default values are:

cubicweb.auth.authtkt.session.hashalg = sha512
cubicweb.auth.authtkt.session.cookie_name = auth_tkt
cubicweb.auth.authtkt.session.timeout = 1200
cubicweb.auth.authtkt.session.reissue_time = 120
cubicweb.auth.authtkt.session.http_only = True
cubicweb.auth.authtkt.session.secure = True
Persistent Authentication

This is also a AuthTktAuthenticationPolicy. It is used when persistent sessions are activated (typically when using the cubicweb-rememberme cube). You may overwrite default configuration values by adding configuration entries using the prefix cubicweb.auth.authtkt.persistent. Default values are:

cubicweb.auth.authtkt.persistent.hashalg = sha512
cubicweb.auth.authtkt.persistent.cookie_name = pauth_tkt
cubicweb.auth.authtkt.persistent.max_age = 3600*24*30
cubicweb.auth.authtkt.persistent.reissue_time = 3600*24
cubicweb.auth.authtkt.persistent.http_only = True
cubicweb.auth.authtkt.persistent.secure = True

Warning

Legacy timeout values from the instance’s all-in-one.conf are not used at all (`` http-session-time`` and cleanup-session-time)

Secrets#

There are a number of secrets to configure in pyramid.ini. They should be different one from each other, as explained in `Pyramid's documentation`_.

For the record, regarding authentication:

cubicweb.auth.authtkt.session.secret

This secret is used to encrypt the authentication cookie.

cubicweb.auth.authtkt.persistent.secret

This secret is used to encrypt the persistent authentication cookie.

cubicweb.pyramid.auth.includeme(config)[source]#

Activate the CubicWeb AuthTkt authentication policy.

Usually called via config.include('cubicweb.pyramid.auth').

See also cubicweb.pyramid.defaults

class cubicweb.pyramid.auth.UpdateLoginTimeAuthenticationPolicy[source]#

Bases: object

An authentication policy that update the user last_login_time.

The update is done in the ‘remember’ method, which is called by the login views login,

Usually used via includeme().

cubicweb.pyramid.bwcompat#

cubicweb.pyramid.core#

Binding of CubicWeb connection to Pyramid request.

cubicweb.pyramid.core.includeme(config)[source]#

Enables the core features of Pyramid CubicWeb.

Automatically called by the ‘pyramid’ command, or via config.include('cubicweb.pyramid.code'). In the later case, the following registry entries must be defined first:

‘cubicweb.config’

A cubicweb ‘config’ instance.

‘cubicweb.repository’

The correponding cubicweb repository.

‘cubicweb.registry’

The vreg.

cubicweb.pyramid.core.repo_connect(request, repo, eid)[source]#

A lightweight version of cubicweb.server.repository.Repository.connect() that does not keep track of opened sessions, removing the need of closing them

cubicweb.pyramid.core.get_principals(login, request)[source]#

Returns the group names of the authenticated user.

This function is meant to be used as an authentication policy callback.

It also pre-open the cubicweb session and put it in request._cw_cached_session for later usage by _cw_session().

Note

If the default authentication policy is not used, make sure this function gets called by the active authentication policy.

Parameters

  • login – A cubicweb user eid

  • request – A pyramid request

Returns

A list of group names

cubicweb.pyramid.core._cw_session(request)[source]#

Obtains a cw session from a pyramid request

Parameters

request – A pyramid request

Returns type

cubicweb.server.session.Session

Not meant for direct use, use request.cw_session instead.

cubicweb.pyramid.core._cw_cnx(request)[source]#

Obtains a cw session from a pyramid request

The connection will be commited or rolled-back in a request finish callback (this is temporary, we should make use of the transaction manager in a later version).

Not meant for direct use, use request.cw_cnx instead.

Parameters

request – A pyramid request

Returns type

cubicweb.server.session.Connection

cubicweb.pyramid.defaults#

Defaults for a classical CubicWeb instance.

cubicweb.pyramid.defaults.includeme(config)[source]#

Enable the defaults that make the application behave like a classical CubicWeb instance.

The following modules get included:

It is automatically included by the configuration system, unless the following entry is added to the Pyramid Settings file:

cubicweb.defaults = no

cubicweb.pyramid.login#

cubicweb.pyramid.profile#

cubicweb.pyramid.session#

Web session when using pyramid#

CubicWeb CWSession entity type so that sessions can be stored in the database, which allows to run a Cubicweb instance without having to set up a session storage (like redis or memcache) solution.

However, for production systems, it is greatly advised to use such a storage solution for the sessions.

The handling of the sessions is made by pyramid (see the `pyramid's documentation on sessions`_ for more details).

For example, to set up a redis based session storage, you need the `pyramid-redis-session`_ package, then you must configure pyramid to use this backend, by configuring the pyramid configuration file:

[main]
cubicweb.defaults = no # we do not want to load the default cw session handling

cubicweb.auth.authtkt.session.secret = <secret1>
cubicweb.auth.authtkt.persistent.secret = <secret2>
cubicweb.auth.authtkt.session.secure = yes
cubicweb.auth.authtkt.persistent.secure = yes

redis.sessions.secret = <secret3>
redis.sessions.prefix = <my-app>:

redis.sessions.url = redis://localhost:6379/0

pyramid.includes =
        pyramid_redis_sessions
        cubicweb.pyramid.auth
        cubicweb.pyramid.login

Warning

If you want to be able to log in a CubicWeb application served by pyramid on a unsecured stream (typically when you start an instance in dev mode using a simple cubicweb-ctl pyramid -D -linfo myinstance), you must set cubicweb.auth.authtkt.session.secure to no.

Secrets#

There are a number of secrets to configure in pyramid.ini. They should be different one from each other, as explained in `Pyramid's documentation`_.

For the record, regarding session handling:

cubicweb.session.secret

This secret is used to encrypt the session’s data ID (data themselved are stored in the backend, database or redis) when using the integrated (CWSession based) session data storage.

redis.session.secret

This secret is used to encrypt the session’s data ID (data themselved are stored in the backend, database or redis) when using redis as backend.

cubicweb.pyramid.session.includeme(config)[source]#

Activate the CubicWeb session factory.

Usually called via config.include('cubicweb.pyramid.auth').

See also cubicweb.pyramid.defaults

cubicweb.pyramid.session.CWSessionFactory(secret, cookie_name='session', max_age=None, path='/', domain=None, secure=False, httponly=True, set_on_exception=True, timeout=1200, reissue_time=120, hashalg='sha512', salt='pyramid.session.', serializer=None)[source]#

A pyramid session factory that store session data in the CubicWeb database.

Storage is done with the ‘CWSession’ entity, which is provided by the ‘pyramid’ cube.

Warning

Although it provides a sane default behavior, this session storage has a serious overhead because it uses RQL to access the database.

Using pure SQL would improve a bit (it is roughly twice faster), but it is still pretty slow and thus not an immediate priority.

It is recommended to use faster session factory (pyramid_redis_sessions for example) if you need speed.

cubicweb.pyramid.url_redirection#

cubicweb.web.views.urlrewrite#

cubicweb_web - And how to use it with CubicWeb#

cubicweb_web is a cube which used to be part of CubicWeb. It provides a lot of generic HTML views and a controller mechanism.

First steps#

Routing#

CubicWeb offers two different ways of routing : one internal to CubicWeb and a one with the pyramid framework.

  • Principle:

    • cubicweb and pyramid

    • the CW request object

    • the pyramid request object

    • encapsulation of the CW request in the pyramid request

    • bw_compat and the options to use, fallback when CW doesn’t find anything

  • CubicWeb routing:

  • Pyramid routing:

    • general principles

    • predicates

    • tweens

    • content negociation

Front development#

  • With Javascript / Typescript (using React):

    • general principle

    • how to install and integrate js tooling into CW

    • cwelements

    • rql browser

  • With Pyramid:

  • With CubicWeb Views:

  • RDF:

    • the RDF adaptator

    • RDFLib integration into CW

Security#

Cubicweb configuration files#

Common Web application tools#