Designing MyBB 1.9’s Installer

One key to keeping software projects and the surrounding communities healthy is keeping friction for all audiences to a minimum.

In MyBB, this friction is derivative of user experience and developer experience. Our largest audience is formed by the end users — people browsing online forums, not expected to know what MyBB is, yet benefitting from fine-tuned visuals, phrases, and flows that come out-of-the-box. At the same time, we target two groups further down the forum assembly line, for whom both UX and DX apply.

For site owners and community leaders, the software needs to be approachable and intuitive — without requiring particular knowledge of languages and technologies — but also allow tweaking its look and functionality by maintainers with technical experience.

For developers, in addition to a useful extension system, APIs, and documentation, the software needs to expose the appropriate tools to allow speedy development and testing — without assuming one’s familiarity with it.

These factors are crucial in the world of free and open-source software, where the development relies on external contributors and their ease of work.

A setup mechanism is where their paths cross: it has to break down unavoidable complexity, without getting in expert users’ way. Besides having to meet best UX and DX practices, it also carries the weight of defining the first impression of the product for everyone.

The Need for Speed

kawaii — 2:56 PM

I wonder how many of the PostgreSQL installs are me with my Docker stack

People who work with, and on MyBB, install it a lot. To comfortably test new code and eliminate bugs in the core and extensions, their setup should require minimal time and attention better spent on the task at hand.

The existing installation experience left much to be desired — among others, the old installer:

A screenshot of the Table Creation page displayed during the installation of MyBB 1.8, with an unnecessarily long list of names of created tables.
              • is strictly synchronous and static, making users alternate between waiting and filling out forms,
              • asks for information that’s either nonessential (e.g. a website URL for the optionally displayed link), or derived (e.g. cookie settings that can be deduced from the forum URL),
              • contains technical details of little to no relevance, which also makes it more difficult to navigate,
              • loads pages only for the user to press Next, instead of proceeding automatically,
              • offers no shortcuts for quick setup for testing or development, and
              • can’t be scripted or automated.

The special part of the application accessed through install/ was largely self-contained and separate from the rest, offering a good target for improvements parallel to other work on the 1.9 series.

In this post, we share how the system was disassembled, redesigned, and rebuilt.

Continue reading

1.8’s CAPTCHA Bug, Parser Validation, and PHP Compatibility

As we stabilize the 1.8 branch for future support with development eventually switching focus to 1.9, we’d like to draw your attention to the following advisories.

CAPTCHA Bug

Version 1.8.27 has introduced a bug affecting two of MyBB’s supported CAPTCHA mechanisms: reCAPTCHA v3 and hCaptcha invisible. For those, the CAPTCHA may appear broken, and the verification can reject or accept attempts incorrectly.

If your forum uses those systems, we advise to either:

  • temporarily switch to another mechanism using the CAPTCHA Images for Registration & Posting setting (ACP: Configuration → Settings → General Configuration), or
  • applying the upcoming changes to source code files manually.

This problem will be resolved in the next maintenance release.

Parser Output Validation

The upcoming maintenance release enforces validation of XHTML code generated by the MyCode parser in order to improve security.

MyBB 1.8.27 included this feature in report-only mode, meaning that any problems are already being saved to the configured error log. After upgrading, validation errors will continue to be logged, but messages with problematic MyCode will not be displayed to prevent potential XSS attacks against your forums.

Forum administrators should verify that their error logging is configured properly, and monitor the log for errors that may indicate necessary changes to their customizations like custom MyCodes, theme templates, username styles, and plugins. These errors can be triggered when forum content that uses MyCode is viewed.

We created a relevant Docs section that details pinpointing the origin, debugging using a dedicated tool, and disabling the validation requirement for boards that are not yet ready for this change.

Examples of Fixed Validation Errors

To help demonstrate what actions may need to be taken, let’s take a look at some validation failures that turned up so far:

  • Case 1: Attributes Without Value in Default Templates

    MyBB’s default theme included HTML attributes without values. These caused validation errors such as:

    • Specification mandates value for attribute
    • attributes construct error

    To fix this, we simply added ="true" fragments where needed.

  • Case 2: Redundant Tags in Username Style

    In a support thread, unnecessary HTML in a customized username style, present in a forum post, resulted in a logged failure that mentioned:

    • Opening and ending tag mismatch

    • Extra content at the end of the document

    This could be resolved by cleaning up the HTML code in the Username Style field for the problematic user group by removing stray closing tags.

  • Case 3: Self-Closing Tags in Custom MyCode

    In another support thread, a custom MyCode included an unclosed <hr> HTML tag, which resulted in a validation failure that mentioned:

    • Opening and ending tag mismatch

    • EndTag: '</' not found

    For correct XHTML validation, tags that don't have an equivalent closing tag should include a forward slash: <hr />.

  • Case 4: Invalid Placeholder Format

    A plugin that inserted invisible markers in the <example#0> format resulted in errors referring to attribute parsing and missing end tags. This format was changed to <example id="0" /> to pass the validation.

If you have trouble resolving validation failures, visit our support platforms and include the full logged error.

PHP Compatibility

MyBB aims to support most recent versions of web browsers, servers, database systems, and PHP interpreters. Due to significant changes in PHP 8.0, however, we recommended using PHP up to 7.4 while the code was being adjusted.

The upcoming MyBB release includes another batch of such adjustments, and removes some unnecessary side-effects of version-related PHP Warnings. We also pay attention to PHP 8.1, which is not expected to cause major problems after these updates.

Even though more issues may still be discovered when running MyBB on latest versions of PHP, we encourage administrators and extension developers to verify the stability of their forums and extensions on PHP 8, and to watch out for any errors that may appear in the error log, starting with the next maintenance release. Numerous web hosts already support switching to PHP 8.0, and MyBB can easily be tested locally using Docker.

Any suspected issues related to compatibility, as usual, can be reported on our support platforms.

Looking to the Future

As we near the release of MyBB 1.8.27, we’re starting to look towards the future of the Project and where we’re headed. In this post, we’ll lay out our plans going forward.

1.8.27 Is a Big Release

The upcoming 1.8 maintenance release is shaping up to be the second biggest in the series, with over a hundred Issues already resolved.

Among others, we have changes to how the mail queue is processed, the addition of some new PDO based database drivers for MySQL and PostgreSQL, some additional plugin hooks, pagination added to some ACP modules, an alteration to exclude bots and spiders from increasing thread view counts, enhanced the attachments upload user experience, and much more!

We’ve also added some other quality of life enhancements behind the scenes, such as some automated tools to check PHP syntax for all Pull Requests and Commits to the GitHub repository and some improvements to our support for PHP 8.0.

The release has been a long time coming, but it’s now just around the corner. I’d like to take this opportunity to thank all of our wonderful contributors, and to ask a favour: if you can, please test the current code as much as you can! We want to make sure that 1.8.27 is a rock solid release. As usual, once the final Pull Requests are merged, you will find a pre-release thread in the 1.8 Development forum.

The Future of 1.8

With 1.8.27 being such a large release, we’ve been thinking about the future of where MyBB is headed.

As such, we’ve decided that MyBB 1.8.27 will be the last major release of MyBB 1.8.

From this point forwards, the 1.8 series shall only receive security fixes and bug fixes for critical bugs that break core functionality.

The reasoning for this change is simple: we need to focus all of our limited resources on one single task – namely, getting MyBB 1.9 released.

MyBB 1.9

MyBB 1.9 is something we’ve been talking about for a long time (too long, a lot of us would say).

We’ve been working on it side-by-side along with supporting the MyBB 1.8 series, which has unfortunately meant the new release has only had limited attention paid to it.

With 1.8.27 being the last big 1.8 series version, all attention will now be turned to 1.9. There will be a freeze made to the merging of any Pull Requests to the project for a period of roughly two weeks in order to finalise the rebase of MyBB 1.9 to incorporate all of the changes present in 1.8.27.

Once this is complete, attention will turn to the following tasks:

  • Scrutinising all new templates to ensure that all changes made to 1.8 in recent releases are reflected in the new templates.
  • Tracking down any remaining usages of the old $templates based code for templates.
  • Writing the ACP management module for the new template system.

Once these tasks are complete, we’ll be at the stage of beginning testing the release in full. At that point, we’ll put a demo install online for everybody to play with, which will reset every day at midnight. This should give everybody a chance to help us debug the release and polish it up.

An Apology and a Thanks

On a final personal note, I’d like to apologise to the Community for the severe lack of progress with the Project and communication from us.

When I joined, forums were booming and MyBB in particular was abuzz with activity. We had a large bustling Team with members from all over the world contributing many changes and improvements. I’ve watched the Project go from MyBB 1.2 to 1.4; from 1.4 to 1.6 and 1.6 to 1.8. Over that time, things have changed a lot! The rise of social media and smartphones have changed the landscape of internet communities significantly.

Unfortunately, with these changes we’ve seen quite a decline in the progress we’ve made with the Project recently. I wish we had an easy fix to this and we could go back to the activity levels that we’ve seen before, and if anybody has any concrete ideas we’d be very happy to hear them in a constructive manner.

I’d like to take the opportunity to thank everybody who has stuck with us over the years and contributed in any way — be it via financial support on OpenCollective; via bug reports; via Pull Requests; via providing support to other members of the Community; or via any other means. Without you, MyBB simply would not exist.

A Close Look at 100+ Patched Vulnerabilities

With the release of MyBB 1.8.22, over one hundred vulnerabilities have been addressed in the 1.8 branch. In this post we look into what the numbers can tell us so far, and how the trends are expected to change in the future.

Since the announcement of the first stable package in 2005, over 270 security flaws were fixed in the 87 versions starting with MyBB 1.0 (some of which overlapped when two supported branches received security updates simultaneously), making 88% of all releases related to security fixes. Additional information tracked within the 1.8.x branch surfaces the most common means of reporting, types, and practical impact of 103 vulnerabilities that have been addressed five years in.

Continue reading

MyBB Merge System Support Update

Some of you may be aware back in October 2016 with the release of MyBB 1.8.8 & Merge System 1.8.8 we ended the support for the current MyBB Merge System. The plan at that time was to rebuild the system from the ground up alongside the release of MyBB 2.0. As MyBB 2.0 has been put on hold indefinitely and a growing number of users are experiencing problems using the Merge System, I am pleased to confirm we are planning to reverse that decision and continue supporting the existing Merge System for the foreseeable future.

The current repository remains on GitHub and we will be using Git as normal to track issues and releases. The current plan is to release the next version with MyBB 1.8.23. A number of PR’s have already been merged since the official end of support in 2016 including bug fixes for PHP 7.2’s compatibility and support added for XenForo 2 . Please bear with us while we try to get on top of the issues reported some time ago and re-familiarising ourselves with the system again.

On a side note, Happy New Year to you all and we hope 2020 is a success.

 

Thanks,

MyBB Team

MyBB 1.9 Development Update

With the continuous Community effort to improve the quality of our stable branch in the background, the work on remaining features of MyBB 1.9 moves on.

One of the completed changes that will make its way into MyBB 1.9’s highlights — other than the theme system — is the introduction of modern password hashing. The md5-based hash function, used in MyBB since its very beginning, will be replaced by bcrypt, making it much more difficult to obtain original passwords basing on new hash values in case of a data breach.

The current blocking task for other areas is rebasing the 1.9 branch on top of the 1.8 branch. Due to the way that Git (the tool we use to manage development) works, 1.9 is worked on in a separate branch whilst 1.8 development progresses. As the 1.8 branch moves forward, the 1.9 branch slowly goes out of sync, missing the recent changes from 1.8.

At the current moment, the 1.9 branch is in line with MyBB 1.8.17, leaving us with 4 versions’ worth of changes that we need to merge into 1.9. This, unfortunately, is not an easy process due to the nature of code changes in 1.9 and there are a lot of conflicts which need to be resolved manually. To ease this process we have decided to rebase one version at a time. Some MyBB 1.8 releases contain a smaller amount of changes than others, and these are considerably easier to rebase too.

Once the re-base is complete, there are still a couple of other tasks for 1.9 before we can release our first Alpha and Beta releases. Some of these issues include:

  • Implementing a new email system. The current MyBB email system causes no end of support threads due to its limited support for slight variations in the way email servers “speak” the SMTP protocol. We’re proposing that we adopt an existing well tested and support email sending library to manage the sending of emails. From a core point of view, this should be relatively simple since almost every email sent uses a single standard function (my_mail()).

    We’re proposing that we adopt the new symfony/mailer library which provides easy ways to send email via SMTP as well as various email APIs such as Postmark.

  • Reviewing any missed templates from the Twig conversion. During the rebase effort, we’ve noticed some lingering uses of the old template system within the core. These need to be rounded up and eliminated to ensure the template system usage throughout is consistent.

  • Updating the ACP to allow editing of Twig template files. So far, the Team have all been editing Twig templates directly through their respective files. While this is a great way to work (who doesn’t want to use their own editor of choice?), being able to edit templates easily within the Admin Control Panel is a useful feature that needs updating to work with the new template system. There is some discussion about looking at the JavaScript code editor that we use when editing templates to see if there are any better options on the market. An often requested feature has been the ability to edit multiple templates within tabs at the same time, an enhancement which would be very handy when working with new templates.

    We’re also looking at the possibility of leaving certain level of support for the old template format to reduce the number of changes required in MyBB 1.8-based plugins to work with MyBB 1.9.

    Open the 1.9 Theme System Issue issue to see what design problems we’ll be aiming to solve, and to participate in the discussion, whether you’re an Extension guru or have previously noticed friction when dealing with themes in MyBB.

    View on GitHub

We will also be starting to update and introduce documentation for 1.9. If there are any documentation pages that you would like to see updated or improved, now would be a great time to bring them to our attention!

Blueprinting Automatic Updates for PHP Applications

Keeping MyBB boards secure is a team effort. Security issues discovered and reported by external researchers and our core developers are analysed, fixed and included in final packages. The process doesn’t end there however: it is essential that administrators are notified to update their forums as soon as possible in order to prevent the addressed vulnerabilities from being exploited in an attack on their boards and users.

Learn More

Our recently published summaries, recommendations and links to reviewed guides in the SECURITY.md file contain many resources forum administrators can use to secure their boards against both opportunist and experienced digital criminals. First and foremost though, we always recommend that users keep their MyBB installs up to date. We also suggest using the new subscription feature for all used plugins and themes on Extend.

Based on our experience, even large discussion boards that don’t have dedicated technicians tend to use outdated versions of MyBB and the situation in the area of extensions might be equally concerning. Not unlike other software, periodical updates are the main method of delivery for security patches — most MyBB releases contain fixes plugging security holes ranging from theoretical risks to critical vulnerabilities.

Issues Addressed in MyBB 1.8.x by Version

The need for continuous response to vulnerability reports is a strong argument for making the reduction of manual effort needed to keep our packages up to date a long-term goal.

In this post we’ll explore what keeps our developers up at night that also affects MyBB’s ability to introduce automated updates, and how the mechanism might be actually implemented once the system — currently being rebuilt for version 1.9 and subsequent branches — is ready.

Continue reading

Building software packages with Docker and Phing

Every meaningful set of development activity in open-source projects like MyBB is followed by an official release that merges in additional lines of production, like security updates, and wraps it up with descriptions and instructions easy to understand for non-developers and site maintainers. Currently the most popular way of distributing updates to PHP-based software is file packages: project managers have to scramble to gather and bundle all files and associated documentation while site administrators are expected to keep track of (and sometimes interpret) this information.

This, to convenience of site administrators and ours, is planned to improve upon adoption of concepts like continuous integration that put emphasis on making all products deployable after every change to the code, and the integration of tools like Composer, which ease the pains of managing third-party solutions and allow to separate one big product into small, handy modules. Even though conveniences like fully automated updates will take time to become reality with informal open-source projects (where the technicalities are much easier to implement than procedures that provide a reasonable level of security), MyBB moves closer to that with eliminating manual tasks covering a broad range of activities that precede each release—the last 4 versions of MyBB (starting with 1.8.12) have been build using the recently published package builder.

The MyBB build repository

Rewriting Memos in XML & PHP

The core part of the builder’s logic is Phing, an Apache Ant-based PHP task build system. This engine enables developers to specify operations related i.a. to git & patch (extensively used to apply sensitive patches before the release), file encoding and archiving saved in an XML build file. It’s also used to call sub-scripts that list changed files and calculate archive size and checksums, but also perform some project-specific operations like counting modified language files, searching for templates that changed between versions or update plugin hook locations with line number precision. Since the Jekyll-powered MyBB.com website is generated from Markdown & Front Matter data files, the builder also prepares the version’s YaML metadata ready to be put into the repository allowing Release Notes and the Release Blog Post content to be generated.

You Want to Run it on What?

Another important role plays Docker, a platform introducing container systems. You might recognize it from the recently put out image recipe that can be used to deploy MyBB 1.8, however this environment is also used whenever packages need to be assembled. No matter who, where or when participates in the building process, they should be able to use the same precisely defined tools—by running the script inside a container we can assure a degree of confidence in that, given the separation from the host operating system. Our Docker image, based on a trimmed down version of Debian, contains an unsuspicious development toolset including basic packages and a PHP interpreter with customized configuration and the strip-nondeterminism tool that normalizes the output to make it possible to arrive with byte-to-byte equal archives identified by matching checksums. This practice is called build reproducibility which will serve as a vital part in download verification.

Real output (with real git errors) when building MyBB 1.8.15 packages

Visit the mybb/mybb-build repository to set up own production line basing on our code and compare against the latest MyBB release packages (starting with 1.8.15, releases on GitHub include a build package with input necessary to reproduce the output).

The MyBB build repository

Automated packaging does not only leave more time for other aspects of running large-scale projects, but also assures that every update is brought to users without potential mistakes that could have been made otherwise with manual assembling. Furthermore, whenever mistakes are spotted, the archives can be quickly rebuilt and pushed out—less emphasis will be put on singular releases and more on their continuous delivery with seamless upgrades that MyBB will be working on.

Shuttering of the 2.0 host compatibility repository

Not long ago, we started a new project to gather information about the PHP versions that various web hosts support. The aim of this project was to gather a list of web hosting companies who would be able to host the upcoming MyBB 2.0 release.

Since starting that project, we have received several contributions. However, we’ve also since changed the PHP version requirements that we will require for 2.0. As such, we are closing the mybb/2.0-Hosts repository and instead asking that users please make use of the PHP Versions website to track the PHP versions on offer at different hosts. This website operates in a similar way to the old MyBB project, but is much more widely used and already contains information for lots of different hosts. For more information on contribution to the PHP Versions website, please see the contributing guide.

MyBB 2.0 Repositories Are Finally Public

Yep, you read that right. The MyBB 2.0 repositories are finally open to the public for browsing and contribution. The repositories we are opening are:

  • mybb/mybb2 – the core of MyBB 2.0, based on a Laravel skeleton template. This repository contains the core forum and is where the majority of work happens. The overall structure of this repository will be changing through the New Year to make deployment easier on shared hosts and other setups.
  • mybb/Auth – a modified authentication package for MyBB 2.0. This is based around the Laravel 5.0 Auth package, and is about to receive a major overhaul to make use of the Laravel 5.1 and 5.2 authentication changes.
  • mybb/Parser – the core post parser for MyBB 2.0. The majority of the parsing logic has simply been imported from MyBB 1.x in the current incarnation in order to get a working parser implementation. The future plans for this repository include adding further unit tests and refactoring the BBCode/MyCode parsing to use a proper parser/lexer rather than a large jumble of regular expressions.
  • mybb/Settings – the core settings system of MyBB 2.0. This package provides both site-wide and user settings and can easily be used for other projects and plugins. Settings are split into “packages”, with “mybb/core” being the core MyBB package. Further documentation and details will be explained about this package in an upcoming development post.

Browsing these repositories should make it fairly clear that while MyBB 2.0 has had a good start so far, it is still far from done. Original plans were to have an Alpha release available by the end of 2015, but these plans have unfortunately not come to fruition. It is our hope that making these repositories public will bring more contributions, suggestions and feedback from you the community.

For now we are simply opening these repositories, but over the New Year and Christmas holidays we will be documenting our code formatting, contribution guidelines and our roadmap and plans for MyBB 2.0 more fully in the MyBB 2.0 Planned Features forum. Rather than blogging about each of these, we will be writing topics that will be stuck to that forum to provide guidelines about our expectations.

We also plan to keep up our (recently lacking) development blog posts which will cover the usage of the components we are open sourcing as well as other components that will be created in the future. Future components and progress will be developed fully in the open, utilising our new BSD-3 licence.

As a closing note, MyBB 2 is in no way stable yet and should not be used on a live forum at all right now. Open sourcing these repositories is aimed primarily at developers and experienced administrators wanting to provide input and feedback on the future direction of MyBB. No support will be provided for any of the code in these repositories until we reach a Beta release.

We wish all of our users a happy holiday season,

The MyBB team